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ABSTRACT " > 

This report presents a detailed rationale and a plan ' 
for a study of the effectiveness of materials and procedures for 
meeting the educational needs of individual children in compensatory 
education programs. An introduc torV^section . addresses an overview of 
the policy questions to which tha/results of the individualized 
instruction study will be relevant, au outline of the di-stinguishing 
features of the proposed approach, and a discussion of alternative 
diesigns t.hat will serve to further clarify the_ design that is • * 
recommended.. Other sections include foci on defining and assessing 
program' and support variables/^assessing outcomes, sampling and 
securing the cooperation of schools, collecting data, reducing and 
analyzing data, project organization, staffing, schedule and^ 
references. • Appendices include in-^class interviewing, analyzing 
curricula^, behavioral analysis of curricula, videotaping^ and test 
reviews, ihe outcome of .primary interest in this study is 'stated to 
be achievement in readin.g and mathematics, other outcomes also having 
been considered for inclusion. The assessment section describes the- 
•outeome measure that is recommended for assessing student achievement 
and the ratio%a^e for its selection. In addition, measurement of 
program effecl^^on noncognitive student outcomes and on the total 
classroom efivff^nment- is' discussed. (Aathor/AM) 
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' ' '* ^ Section 1 ' 

■ ^ ' Introduction • 

Congress has directed, the National Institute of Education (NIE) to 
perform a general examination of compensatory education programs, with 
the specific instruction to conduct at^etailed study of the effectiveness of 
materials and procedures for meeting the educat'ional needs 9f individual 
children. This report presents a detailed rationale and plan for the con- 
duct of such a study. Included in this introductory section of the rejDoft 
are an overview of the policy questions to which the results of the Individ- 
ualized Instruction Study will be relevant, an outline of the distinguishing 
features of our propos^ed approach, and a discussion of alternative designs 
that will 'serve to fi^rther clarify the design that we are recommending. 

Policy Questio;ns . - ^ 

The findings of the Individualized Instruction Study will be rele- 
vant to policy questions that have 'been .of major concern to Congress, to ^ 
NIE, 'and tc^ others interested in education. 'The overall question concerns 
the success with which various Educational approaches are compensating 
for children's initial educational disadvantage by meeting th,eir individual 
needs. Related to this question are two major issues. The first concerns 
the ways in which compensatory education funds can be most effectively 
utilized. Should funds be provided for supplemental services and materials 
for disadvantaged children, should they be provided for specific programs 
in which only students labeled as educationally disadvantaged can partici- 
pate, or should an alternative allocation procedure be adopted? The sef- 
ond issue is whether or not compensatory education funds should be pro- 
vided at all. If it is true that schools don4 affect student learning, as sev- 
eral large-scale studies have suggested, then why continue. to spend more 
and more money on educational programs, compensatory or otherwise. 
-The design that we ^^re recommending has been formulate&-with these is- 
^sues in mind. % ^ 



To better understand the relevance of the study proposed here to 
the first of these policy issues, it is useful to review the procedures 
followed in Title I of the Elementary axid Secondary JSducation 'Act, 
revealed by our informal examination of programs' in Pennsylvania and 
New-York. Title* I is the most massive Federal effort for the support 
of compensatory education. Within a school district, the allocation .proc- • 
ess begins with the. collection of information on the socioeconomic status 
oft families living in the attendance area for a particular school. That - 
schooTs eligibility for Title I funds is a function of the number of chil- 
'dren from low income families within its attendance area in relation to 
the number of economically disadvantaged children from other attend- 
ance areas in the s chool district. Generally, any school having a per- ' 
centage of low income childrBn equal to or exceeding the district- wide 
percentage i& eligible for Title I funds. ^ . 

However, within a school designated as eligible, only those chil- 
dren who are defined as educatiqnally disadvantaged may actually benefit 
from Title I funds. li^ucationally disadvantaged children are those who 

' Another important policy issue is whether funding, even- if supplemental, 
should be based on economic disg^dvantage or educational disadvantage. 
It would seem that this question can be settled most easily by establish- 
ing the relationship between economic indicators and educational achieve- 
ment indicators. Reviewing data for two ur^an school systems, (Pittsburgh 
and Minneapolis), we found that correlations between two economic indi- 
cators (percentage of families receiving Alitt for Dependent Children funds > 
and percentage of families earning below $2, 000) correlated in the range 
.75 to . 95 with the percentage of children in a school scoring at least one- 
half year below grade level, using the school as the unit of analysis. Thus, 
schools would receive essentially the same suppo'rt whether funding is 
based on economic or achievement indicators. Economic indicators seem 
to be the preferred political solution because the distribution of funds can 
be estimated in advance and because more members of Congress ^are will- 
iiXg"to view Title I support as a poverty measure. Also, distribution based 
on economic indicators eliminates the negative incentive to improve achieve- 
ment, wl;iich is built into allocation procedures that are keyed tcJww achieve- 
ment. ^ 



need special assistance in order to raise their level of" educational attain- 
ment €o the level appropriate for children of their ^ge. This group in-, 
eludes childrea whose needs for assistance result from poverty, neglect, 
delinquency, br cultural or linguistic isolation from the community at 
large. ' Most frequently, educationally disadvantaged is operationally de- 
fined as a certain grade level behind one's classmates on standardized . 
achievement tests administered by the school system in the fall. 

Title I funds are administe^ec^ by local education agencies (LEAs). 
At present, the LEAs must demonstrate that the funds are indeed being 
used f6r programs that are supplemental to standard classroom insttfuc-\ 
tion. Title I programs in the lower elementary schoj^l often involve the 
hiring of a special tutor who works ^yith those child^n labeled as disad- 
vantaged. Tutoring generally occurs in a corner of the classroom or ^n 
a separate room while the regular teacher is teaching something else to 
the rest of the class- -instruction that the disadvantaged children would 
not have thApportunity to benefit from. The confusions and inequities 
that the present allocation process produces are even more exaggerated 
when, one considers a classroom where a majority of the students in the 
room are eligible for Title I funds as define Ay their low achievement 
scores. How does one supj^lement a classroom program, with Title I funds 
.when 70 percent of the students in the room are eligible? One approach 
is for the school to segregate students on the basis of achievement scores 
and have an entire classroom of all eligibles. 

It seems that it would be far more effective if Title I funding could 
be used to subsidize a strong basic program for all children in schools . 
that include some specific percentage of children designated as eligible 
for special assistance. A basic'pro'gram that is adaptive to individual dif- 
ferences in all students would probably be more educationally sound than 
the programs currently supported by Title I that are restricted to eligible 
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children! Secondly, an individualized program could eliminate the need 
to brand any particular group gf students as dis.advantaged. One nega- 
tive factor in compensatory programs is that being labeled disadvantaged 
often produces a disadvantage (e. g. , White et al. , 1973). Third, 'it would 
be easier to monitor a core program since, only budgets and allocations 
would need to be examined. For example, it would not be necessary, as 
it is under current guidelines,- to make sure that a classroom aide pro- 
. vided'through Title I support helps only those students in the classroom 
who are designated as eligible. 

A related policy issue to which the results of the Individualized 
Instruction Study will also-te relevant concerns the extent to which low 
performance in basic skills such as reading and mathematics can be. af- 
fected by special educational programs. Major studies such as the Equal- 
ity of Educational Opportunity survey (Colema'n et aL , 1966) and the later 
syntheses of this and related efforts (e.g., Averch et al. , 1972; Jencks 
et al. , 1972) are commonly cited as evidence that iiYhovative educational 
programs do not significantly improve student performance or, at least, 
that different programs do rfot make a difference in what children learn. 
The results of the Individualized Instruction Study can possibly be useful 
in challenging this trend, since what happens at the classroom level is 
to be -investigated. Prio.r -research used school variables that were too 
far removed from the educational program actually being implemented 
in the classroom to have much effect on what students in that classroom 
learned. Differences in variables that might have made a difference at 
the classroom level often "washed out" at the school level, leading to. the 
conclusion that school differences don't make a difference. Allowing this 
.conclusion to stand unchallenged would be to give up on improving educa- 
tion. ^ 

Given the confusion of the current allocation procedure and the 
enormou^s cost of compensatory educ9.tion. Congress has requested detailed 
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information on the administration of compensatory programs, fund 
allocation systems, and student development, NIE has chosen to obtain 
the requested information by funding six major studies, four of which 
relate to student development: a nationally representative survey of 
ongoing programs, a synthesis of results to date, a review of alterna- 
tive designs for compensatory education, and an in-depth review of the 
effects of individualized instruction. The results of this latter study, 
for which we are proposing a design, should not only ''clarify for Congress 
what individualization can and cannot do for education, but also provide 
•a detailed picture of compensatory education at the classroom rather than 
the school level. The contractor should be aware of the context in which 
the Individualized Instruction "Study is being undertaken and the magnitude 
of its potential. 

V 

Overview of Design Feature > . 

The design that w^ are recommending incorporates several dis- 
tinguishing features. These features concern program and implementa- 
tion measures, outcome measures, sartipling procedures, and analysis 
procedures. In addition, a special study is proposed to contrast the ef- 
fectiveness of instructional settings in .which the education of compensa- 
tory children can occur. Each of these features is described briefly 
below. 

* 

Program and implementation variables. Our design assumes 
that classrdoms differ in J:he manner in which and degree to which they 
are adaptive to individual differences in children. Classroom practices 
can vary in how they provide for differences j.n the knowledges and abili- 
ties with which children begin a school year, in how they respond to dif- 
ferences in interest and motivation, in ho\y they adapt to dif^.er.ent rates 
of learning, in whether or not they recognize and dearwith cultural dif- 
ferences, and in whether or not they allow for a'Variety of educational 



goals. This variation is the result of: (1) differences in the instruc- 
tional programs being used in classrooms, and (2) differences in the 
way in which teachers implement any given 'program. The design that 
we are recommending includes a set of ^variables that will be used in 
assessing the actual processes that are taking place in classrooms and 
in determin^fig the relative effectiveness of these processes. The model 
of classroom proces s.es 'that guided the formulation of our design is a 
model for explaining the amoimt of learning that has taken place in a 
class-room during the school year. The assumption is that variation in 
learning is a function of the processes that are operating in that .class- 
room and the initial abilities of children. 

■ " / *' 

We have not formulated a set of criteria to determine whether or 
not a parHU^ular s'chool or classroom' is operating an individualized or stan 
dardized program, and then a second set of criteria to determine how 
well the, school or classroom has implemented that program. Instead, 
we have defined a set of dimensions along which all classrooms differ 
and in terms of which they will be assessed. The dimensions reflect 
both pfog:j^am design differences and iniplementation variation. Those 
classroom dimensions* that are related to aspects of individualization are 
referred to as program variables. Other classroorri dimensions that are 
related to learning but not to individualization are viewed as support vari- 
ables. ' , . ■ • 

Outcome measures . We propose/that commercially available 
achievement tests in reading arid mathematics be used in the Individualized 
Instruction Study. A testi^attery is recommended that is superior to any 
battery that could be built in the time available to the contractor, certainly 
in terms of reliability and the elimination of racially biasing items, and 
probably also in terms of content validity. 
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Sampling procedures. The primary objective of our design is 
to identify effective nxechanisms for improving educational programs, ' 

not to evaluate Title I or any other compensatory programs. Too little 

V- 

is learned ftom contrasts that assume that programs are unique enti- 
ties that can be implemented exactly as they were»designed. Therefore, 
the sampling scheme emphasizes the need to achieve va-riance in class- 
room processes, rather than representativeness of present compensa- 
tory practices. It also assumes that the study can best be accomplished 
by an intensive examination of classroom practices in a restricted geo- 
graphic region, and that littife would be gained from the extra expense 
of national represent'ation. 

Analysis procedures . The analytic procedure that we propose 
is based on the recognition that classroom practices will occur in un- 
controllable combina*fcions in the field. What is needed in such situations 
is a technique for sorting out what can be uniquely attributed to particu- 
lar practices, and what is due to combinations of practices or relation- 
ships between practices and initial abilities of children. 

One specific technique that we fecommdnd is commonality analy- 
sis, which is a variation of regression analysis that allows one to dis- 
tinguish between unique effects and effects that are common to two or 
more of the predictbr variables, where the predictors are both class- 
room process dimensions and children's initial abilities. This tech- 
nique will provide the basis for inferences about the extent to which dif- 
ferences in classroom processes produce differences in student achieve- 
ment, the extent to which classroom process effects are due to methods 

of individualization, and the relative effectiveness of different individ- 
♦ 

ualizing mechanisms. ' 

Instructional setting contrast . The main thrust of the study will 
be to identify effective classroom practices, particularly those related 

11 < 
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to individual differences in children, with the abjective of showing how 
the goals of compensatory education can be achieved without the need 
to label specific children as disadvantaged. To complete the study, it * 
will be necessary to contrast the more frequently used Title I practice, 
which is to provide special tutors for eligible children, with the alterna- 
tive of providing for these children in classrooms that adapt to differ- 
ences among all children. Section 6 of this report outlines an approach 
to this contrast of instructional settings. 

Alternative Designs 

^The design that we are suggesting can, of course, be modified 
•in many ways/ For example, a broader geographic distribution of class- 
rooms could be incorpora|fed into the design so as to reduce the likeli- 
hood that regional difference^ could affect the outcomes of the study. 
Another modification might be to build outcome measures specific to the 
study in an attempt to obtain a better correspondence between program 
objectives and tested outcomes. These modifications, however, could 
be implemented only if the numbcjr of classrooms were reduced, since 
such revisions would increase costs significat\1>ly. We have not included 
these altex-natives in our de sign b ecause we feel tha;t achieving an ade- 
quate number of classrooms is of prime importance, that program repre- 
sentativeness rather than geographic representativeness is essential, aud 
that the commercially available outcome measures that we are recommend 
ing are superior to any measure;,s that could beMeveloped in the limited 
time available. 

There is an additional alternative that wotild be a major departure 
from our basic approach. A consideration. of its features will help to 
further clarify our own design. That alternative requires an ojjerational 
definition of individualization leading to a way of categorizing programs 
as either individualized or not based on program descriptions. A further 

■ ■ 12 ... 
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requirement is a set of procedures for ot)serving classrooms that have 
adopted individualizad and standard programs in order to determine if 
they are well iigapl^fnented. To do so would require measures of hov^ 
well the prag>rams operating in classrooms conform to the design spec- 
ifications of the developers Of those programs. 'Once a set of class- 
rooms that are well implemented versions of both types of programs 
is established, the next task would be to contrast achievement results, 
presumably in a covariance design that somehow took initial differences 
in abilities into account. • 

*The main problem with this approach is that it assumes that within 
the two sets of classrooms there is a homogeneous treatment, and that 
there is a distinct, unique difference between' the two sets. This, of ^ • 
course, is not true because all individualized programs vary in how they 
iAdividualize and how well they individualize what they do individualize. 

r 

Similarly, all standard programs individualize in some ways to some 
degree. Gibbons (1971) put it this way: ^'Identifying programs as indi- 
vidualized conveys so little information afiout them, that any teacher can 
on some grounds claim to^e<individualizing instruction" (p. 15). He 




stantial unless only a| single program were examined. 

We believe that too little would be learned from such*a gross 

contrast of individualized , and standard classrooms and tha^t much infor- 

I* ■ « 

mation would be lost by eliminating the implernentation variation that 

\ , . ^ * 

does exist, since this variation can be capitalized on to shed further light 

on which variations are effective and which are not in achieving student 

learning. The design proposed in this report will provide information 

both'on the]degree to which individualization is effective in improving the 
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also pointed out that "individualized programs vary irt the elements of 
instruction they individualize and the degree of individualization in those ^ 
elements" (p. 54). Ajlthough adding the criterion "well implemented^^ ^ 
could reduce the het'ejrogeneity within treatment, it would still be 'feub- 
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iearning. of all childre/i, particularly those from homes that are disjr 
advantaged in sorrie respect, and on the specific wa.ys in-which it is most 
effective. 

• V 
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. Section 2 

Defininp^ and Assessing Program and Support Variables 

We recommend that, both individualized and standardized class- 
rooms be studied using a common 6et of- variables. These variables, 
which are sensitive to both program design differences- and variation in 
jprogram implementation, will provide information on the processes ac- 
tually taking place in the classrooms under investigation and the influence 
of these processes on student learning. The set of* variables that weVj^o- 
pose includes two types of observable classroom variable s : progr^am 
and support. Program|Variables are related to individu^ization; support 
variables are related to classroom practices that are not unique to indi- 
vidualization. Support variables in this field study serve the' same func- 
tion as control variables in an experimental design in that their inclusion 
in the study will make it possible to reduce the probability of alternative 
explanations for observed program effects. 

The designer of any field research must anticipate alternative 
explanations and develop measures of the variables that they suggest* 
Then, through appropriate data analysis techniques ait<i a convincing mod- 
el of the phenomena under investigation, it is necessary to partition those 
effects that are attributable to the program variables from those that 9.re 
due to the support variables and recognize which portion of tfee effects 
is confounded becaus-'e of correlations among program variables and sup-, 
"port variables. The analytical procedures for doing so are outlined in 
Section 6. * ' ' 

The framework for the study of classroom processes that we are 
proposing is provided by the model illustrated in Figure 2. 1 (Cooley & 
Leinhardt, 1975; Cooley & Lohnes, in press). Thp purpose of this model 
is to explain the variation in student performance that occurs among class 
rooms following an extended period of instru'ction in those classrooms. 
The model specifies thai criterion performance is a function of initial 

' • If) ' 
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"student pe<rfor.m.ance and of certain classroom processes that occur in 
■ ; the interval between the assessment of initial performance and the as- 
sessment of criterion performance* Classroom processes are repre- 

r 

sented by four, constructs that are assumed. to affect the criterion per- 
formance: structure and tolacement, instructional events, optportunity , 

- — ' " 

and motivators . • 

« 

»In terms of the model of classropm processes, program vari- 
ables delate prirnarily to *the structure and placement construct aild 
to the instructional events construct. The support variables are re- 
fleeted by the opportunity and motivators constructs^ The 14 pro- ^ 
gram and support variables that are included under the four cflassroSm 
process constructs are described below, along wi-th the measures and 
data collection tecltfiiques that can be used for their assessment. 

Three data collection techniques are used to assess the process 
variables: interviey^s of the teacher, analysis of the curricula by cur- 
riculum experts, and videotapiag of classroom activities. All thi^'e tech 
niques have/been developed and utilized over the past five years in exten- 

- s - 

■r • 

sive research on the impact of classroom processes on achievement 
(Leinhardt, 1975) and in detailed analyses pf curricula with ^spect to^ 
the degree to which they incorporate principles of instructional theory 
(Holland, 1975). 

A'eacher interviews are extremely useful j.n determining specific 
classroom practices that are followed. In general, teachers attempt to 
provide accurate information, particularly if they do not feel threatened 
in any w^y by the questions asked and if they are aware that fc^low-ups 
will be made to ensure the validity of the information obtained. The fa'^ct 
that the interviews'take place in the classroom also encourages teachers^ 
to be precise in their respons'e^s. ' 
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The purpose of curriculum analysis is to provide detailed informa-* 
tion about the structure and quality of the curriculum materials. It is ex- 
tremely important that the actual materials being used in participating 
classrooms be reviewed to see if they incorporate mechanisms for sup- 
porting individualization. In many cases, the form is present but the 
substance is lacking. Curriculum analysis also helps to cros s -validate 
the information gathered from teachers and provides fine-grained informa- 
tion about the specific value of curriculum-designed instructional ^strate - 
gies. 

Videotaping not only contributes to the cross-validation of teacher 
interview data but-also provides unique information about classroom prac- 
tices, partTcularly the quality of the instructional events. Taping requires 
fewer highly trained observers than in-class observation, eliminates the 
possibility of confounding observer's with sites, and provides a permanent 
record of activities that will make it possible to monitor coding acc^uracy', 
recede ambiguous results, and reanalyze data at a later time using a 
different coding scheme.- 

Inhere are distinct advantages to using this three-pronged proce- 
dure to gain information about^classroom processes. First, information 
is gathered in s^uch a way as to permit careful, reliable analysis in a 
different location. Second, by using three techniques that overlap, much 
of the information collected through one method can be verified h?? at 
least one other method. Finally, permanent records are created that 
will be available for reanalysis using a different statistical approach and 

ai^sdng different questions of the* data. 

^ r - . ' * 

A plan for the collection of data is presented in Section 5. The 
instruments ftbepriselves and directions for obtaining and coding the infor- 
mation required are presented in Appendix A ("In-Class Interviewing"), 
Appendix B-1 ("Analyzing Curricula"), ApfJendix B-2 ("Behavioral 



Analysis of Curricura"), and Appendix C (''Videotaping"). All of the in- 
struments have been designed or substantially modified *for inclusion in 
the Individualized Instruction Study. The specific questions and assign- 
ments that are designed to collect data on the 14 process and support 
variables may be revised -further by the contractor. However, our de- 
sign requires that data be gathered, by one method or another, on all of 
the 14 variables. 

Structure and Placemen t ^ » 

^ Table '2. 1 indicates'' the variables and, measures for the structure 

and placement construct, along with the type of instrumentation that can 
be used to collect information on each measure. As this table indicates, 
the information is obtained through interviews" and curriculum analysis. 
.Videotaping is needed only to confirm interview data regarding the groups 
in which instruction is given. 

The four program variables that are included under the structure 
and placement construct descJribe the structure, organization, and se- 
quencj.ng of the instructional materials, and the procedures for placement 
of the student in the curriculum. These variables arq conce-rned with four 
major questions. " First, is the content of the educational sequence speci- 
fied? Measures of the "specification of objectives" variable that attempt 
to answer this question involve: the clarity of the curricular objectives; 
the specificity of the objectives (are they specific and content oriented, 
or are they general program goals); the frequency of presentation of new 
objectives; and the degree to which the teaching materials match the objec- 
tivjes. Second, are students matched to the curriculum according to their 
abilities and interests? Some of the "matching of students and curriculum" 
measures related to this question concern: the presence of placement, moni- 
toring, and mastery ctssessment procedures in the curriculum; the frequency 
of monitoring; and the range of the frequency of monitoring. The third 
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question with which structure and placement deals concerhs mechanisms 
for making decisions with respect to the sequencing and pacing of instruc- 
tion. Examples of measures for the "sequencing and pacing of instruc- 
tion" variable are: the clarity of the sequencing of the cur ricular mate- 
rials, the person who makes sequencing decisions, the presence of self- 
pacing, and the range of the learning rates of chiMren in the classroom. . 
The fourth question concerns the group in which children most often re- 
ceive instruction (e, g, , entire class, group of four individuals). Meas- 
ures* of the "grouping" variable concern the type and size of groups and 
the frequency of regrouping. 

All four of the structure and placement variables relate to the in- 

/ 

dividualization of instruction. That is, the more specifi'c the objectives, 
the clearer the. mechanisms for matching students and curriculum, the 
more opportunities for variation in sequencing and pacing based on stu-* 
dejit interest and learning needs, and the more individualized the instruc- 
tional setting, the more individualized the learning process will be. 

For specific 'example s of how these variables and their measures 
' relate to individualization, consider the "matching of students and cur- 
riculum" variable. There are fourteen measures for this variable, sev- 
en of which deal with the presence of specific procedure s in the curricu- 
lum or procedures constructed by the teacher that are designed to pro- 
mote individualization by permitting easy matching of the student and the 
curriculum. One of these measurea is "p:^csencc of placement, nionitor- 
ing, and-mastery assessment procedures in curriculum. " If, for in- 
stance, a diagnostic mechanism exists in the curriculum;^ or the teacher 
■ has his/her own informal assessment procedure, then it is more likely , 
that individual students' learning needs will be consistently and accurately 
assessed than if no such mechanism were present. If individual learning 
needs are measured, then there is a greater chance that these needs will 

21 . 
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be met than there would be if no testing were done. Once it has been 
established that a placement test does exist in the curriculum, informa- 
tion must be gathered on the overlap between the test and the curriculum. 
If the. test does not reflect the currviculum content, then the chance of it 
being of value in meeting student needs is very low. But even the pres- 
ence of effective matching procedures is not "sufficient^to guarantee in- 
dividualization. Addilional'measures, such as "per-centage of unique 
assignments," '^average numlfer of days since last test," and "range of 
days since last test" are needed to determine actual teacher practices 
related to these procedures. The more unique assignments made on any 
given day, for example, the greater the chance that students' needs are* 
being- met on an individual basis. 

Instructional Events 

Table 2*2 summarizes the variables and measures for the instruc- 
tional events construct. Variables that are included under this construct 
describe the specific mechanisms, proce(Jures, and actions that place the 
learner in contact with the feedback needed to progress toward a specified 
cornpetency. The variables are of two kinds: interpersonal and curricu- 
lar.- The five interpe rsonal variables ("management information," "cog- 

A ' 

nitive teachings to^ndividuals or small groups," "cognitive teacKing to 
whole clas5," "indirect teacher behavior," and "quality of teaching tech- 
* niques") describe the quality, of interaction between the teacher and the 
student. Interpersonal variables mainly concern the content/ affect, and ' 
clarity of the teacher's instruction and the degree to which the teacher in- 
volves the sttident in active learning. Our assumption is that the greater 
the frequency of appropriate cognitive exchanges between student and 
teacher, the greater the chance that the student's le§,rning needs are be- 
ing met. 
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The curriculq^r variable ("quality of materials") relates to the 
quality of the instructional materials used. This variable deals primari- 
ly witl^ the degree to which th^ ^.ssessment procedure is efficient witK 
respect to the amount of time thctt the prcTcess detains the student from 
instruction,"*? the degree to which it accurately matches the student to the 
ciirriculum, and the degree to which the materials elicit responses from 
students that are directly related to material that the student should 
iQS^rn. 

The instructional events construct includes program variab^s 
rather than support variables because the vast majority of individualized 
curricula are designed on the assumption that teaching sequence and 
learning activities are closely matched to each learner's needs and inter- 
ests. Although many individualized programs do not require students to 
be taught on a one-to-one basis, they do assume that students' needs and 
.^interests are bei^ig assessed on an individual basis and that instruction 
is prescribed based on that assessment* Many standardized curricula 
include aspects of individualized instruction, such as the administration 
of placement tests and posttests, but they do not serve the function of 
matching individual students to the curricular materials. 

As an example of how the instructional events variables and meas- 
ures relate to the individualization of instruction, consider the "quality 
of materials" variable and its measures. These measures ("total num- 
ber of Jippropriately determined responses per child^ " "ratio of appro- 
priately determined respons(j^ to total," "consequence ratio," "predic- 
tive validity ratio," and "discriminability ratio") assume that instruction- 
al materials support learning by arranging conditions under whiob the 
emission and reinforcement of the to -be -learned behavior are highly 
probable. Gicnerally, in teaching, some material is presented to the 
learner; the learner interacts with the material in some covert fashion 
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and then makesran overt public response that can be evaluated. Obvious- 
ly, instructional materials differ in the extent to which overt student be- 
havior is called for. When instructional conditions are appropriate, a 
higher per -child average of overt responses is expect©^ to result in high- 
er student achievement. 

If the student's public response is incorrect, the desired private 
behavior has not occurred and thus has nt)t been learned. Materials that 
generate many errors are not effective teaching materials. Effective 
materials are charac^rized by a low error rate. However, a low error 
rate alone does not indicate that the to-be -learned behavio^ has occurred. 
Effective teaching materials ensure that the learner is correct for the 
right reason. Problems in response contingencies are very common in 
teaching.matei'ials. Consider this anecdote by John Holt in How Children 
Fail (1964): ' . ' 

Later, in music class, the children were asked to touch 
their toes when the teacher played a C. The teacher then 
played a little march, to which the children walked around. 
Every time she came to"a C, slie held'it. Naturally, they 
touched them if any note other than C. was held, and when 
- C w?s played without being held, they ignored it. And. this 
woman thought she was teaching ^hem C! (p. 145) 

In this case, the teacher thought the basics for the students' ."c*orrect" 
responses was the discrimination of a particular sound wave frequency, 
• although she was also providing another basis for such correct respond- 
' ing, a held note. Yet 'her objective was not to teach the discrimination . 
of held notes! A similar contingency problem occurs in,many beginning 
reading programs that introduce new vocabulary words accompanied by 
.'pictures bhat the children easily use to guess rather than read the word. 
Th'e "total number of appropriately determined responses per 
child" is the total number of response opportunities available minus the 
opportunities likely to result in errors and the opB^ortunitiQs likely to^be 
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achieved by inappropriate' behavior. It is expected that the per-child 
number of appropriately determined response opportunities will corre- 
late positively with' achievement. • *Yet it is possible that two classrooms 
with the same raw number of appropriately determined response ojipor- 
tunities per child could provide very different learning environments. 
Classroom A may have 12 per-child response opportunities, *of which 
10 are appropriately determined. Classroom B may have 40 per-child 
response opportunities, of which only 10 are appropriately determined. 
*We, would expect the 30 poorl'y designed response opportunities in Class- 
room B to have an adverse effect on the children's overall performance. 
Thus") in addition to a raw number, a ratio of appropriately determined 
responses to total response opportunities is'included as a measure of the 
'^guality of materials" variable. 

These measures of- appropriately determined response opportuni- 
ties may be used to evaluate all teaching materials in terms of well-es- 
tablished principles frequently associated with programmed instruction: 
(1) individual teaching items should evoke the desired, to-be-learned be- 
havior, and (2) the student should be able to give the required perform- 
ance. 'However, a certain portion of the newer individualized or adaptive 
materials is 'not devoted to teaching, but rather to diagnostic testing. 
These materials contain diagnostic tests that either allow the student to 
skip upcoming teaching material or direct the student to specific needed 
material. Diagnostic test items have a very different function than do 

. teaching items and, thus, require different criteriarfor their evaluation. 
It should be apparent that; if material is to be truly individualized, the 
decisions abou^t what material a student is to. receive or skip must be 
accurately made. Regardless of the structure of the material or how 
* much testing and prescribing is done within a curriculum, that material 

• is not adjusting to indiviid-ual differences if the diagnostic decisions are 

2b 



not valid. ' The '-^tbnsequence ratio,'' predictive validity ratio, and 
»»discriminability ratio'' are measures of how well the individualizing 
features. of a curriculum route the student to needed teaching material 
and away from unneeded mafeerialV , 

Opportunity • 'v - ' 

Tablfe 2. 3 lists the variables and measures for the op^jorturjity 
construct. All three types of instrumentation are used to collect data on 
this construct, although teacher interviews are the primary source of 
information. Opportunity consists primarily of t,wo variables: (1) the 
amount of time a student has to learn the subject matter, and (2) the de- 
gree of overlap between what is measured by the criterion test and what 
is taught in the curriculum. The assumption is that all other things be- 
ing^quaj, a student who has a greater opportunity to learn material as. 
sessed by the criterion will know more of that material. For the pur- 
poses of the proposed study, the opportunity^ariable s are regarded as 
support variables, since classrooms can be expected to vary with respect 
to the amount of t\ie allocated to instruction and the amount of overlap 
between the curricuhm?" and the criterion independent of tfie degree of in- • 
dividualization in the classroom. • • 

There are 11 measures for the "amount of time" variable^^y(fl| 
would seem that the amount of time that a student spends in reading and in 
mathematics could be measured directly and that only these' two measures 
would be required for this variable, Howeyer, our experience indicates 
that it is extremely difficult to obtain accurate information on time spent 
in reading and in mathematics axid that total time spent in learning is a 
more meaningful estimate of opportunity. Although we arc including 
reading and mathematics times as measures because of their intuitive 
appeal and because our experience with these measures may not be 
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generalizable to the Individualized Instruction Study, we are also recom- 
mending that total time spent in learning- related activities be estimated 
from ten ofiher measures, which concern the time spent in school, the 
percentage of time, spent attending to what is^ being taught, the number of 
classmates a student must compete with in order to gain the teacher's 
assistance, and the amount of home vyork assigned. 

The second opportunity variable, "currdcular overlap," represents 
an attempt to answer the question, "To what extent is th^ information on . 
the criterion pleasure taught? " Even with a superior teacher and curricu- 
lum, if what is tested is not taught, then the child will not have an oppor- 
tunity- to learn that material, at least in school.- Curricular overlap is 
estima}:ed in two ways. First, curriculum experts are asked to estimate 
which items on the criterion test are covered by each chapter or unit in 
each text or program used in the classroom. Then the amoUnt of mate- 
ri'al covered by the class is' collected and averaged (for example, out of 
25 chapters in the math text, Grgup I covered 10, Group II covered 15, 
and Group HI covered 22) and the overlap estimated. Second, tfeachers 
asked to indicate those items on the criterion test that (;hey^eel 
taughfeE;«;JIhis actional estimate of curricular overlap willVrovide 
information on test material that was taught even though it was iofi in the 
curriculurii, and on material in the curriculum that was skipp^ by the 
teacher. 

Motivators . ' • 

The motivators construct consists of two sets of support variables: 
(1) variables related to aspects of the curriculum that support and encour- 
-age student learning, and (2) variables related to those aspects of teacher 
behavior (other than explicit instruction) and other interpersonal behavior 
that ^pport learning. While there has be^^ great deal of research on 
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motivation, there has. been a limited amoiint of research on validating 
exactly which elements in ^ classroom motivate students to learn. Ide- 
ally, the study would measure only those variables that are known to 
have a dramatic impact on students' learning. But, given the limited 
amount of time to conduct the study and the limited amount of knowledge 
about effective motivators, we must take our best guess as to which ele- 
ments in the environment, are most supportive of student learning. 

As indicated in Table 2. 4, information regardin^g aspects of the 
curriculum that encourage learning (e. g. , variation of format, speed of 
correction on product) is obtained through teacher interviews and curricu- 
lum analysis. Interviews and videotapes are used to collect information 

on interpersonal motivators, such as the degree to which the teacher uses 

* 

praise and the degree to which he/she encourages the use of games and 
contests., self-evaluation and self-management, and peer tutoring. Pre- 
liminary data indicate that peer tutoring is a positive motivator. It may 
also be influencing the academic achievement of students because peer 
tutors are more effective teachers. However, sifice" analysis of the speci- 
fic technique of peer tutoring is nofe within the scope of the proposed study, 
the presence or absence of peer tutoring will be regarded only as an in- 
terpersonal motivator. The final interpersonal motivator is the amount 
of negatiA^e teacher behavior. This measure is negatively related to 
achievement, but it is still a motivator- -a negative one. By negative be- 
havior, we mean punishing behavior as distinct from correction, which 
need not be presented in a negative way. 

Relating Process ConstructsSto Individualized Ir^truction 

It may be useful at this point to relate our four process constructs 
(structure and placement, instructional events, opportunity, and motiva- 
tors) to theriobjective of the Individualized Instruction Study- -to investigate 
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the effectiveness'of individualized instruction. Our -approach recognizes 
that there is considerable variation in how classrooms will be individ- 
ualized, and that even ^o-^called standardized classrooms will be indi- 
vidualized in some ways to feome degree, depending upon the teacher 
and the program being used. Although such heterogeneity in treatment 
invalidates ANOVA-type experimental designs, our design capitalizes , 
on this process variation by corjverting it into an opportunity to establish 
both the effectiveness of individualized instruction, however it may be 
defined, and the specific individualizing features that gLre most effective 
in promoting student learning, 

In the RFP fot the present project, NIE defined an individualized 
instructioi/al program as one that includes: 

1. Stated performance objectives. 

. 2. Individual diagnosis of instructional needs, and 

. instruction based on that diagnosis, 
c 

3. ''Progress tests that assess mastery at regular 

intervals. 

» # 

4. Rate of progress and remedial instruction based 

on testing results. , 

The extent to which classrooms are operating programs with these features 
will be measured by the variables included in our structure and placement 
construct. If, in ,the data analysis, these variables are shown to be effec- 
tive in explaining gains in reading and mathematics achievement, then 
individualized instruction, as defined by the RFP, is effective. 

A behavioral analyst might define individualized instruction as 

instruction that ^'places the learner in contact with the reinforcement or 
^ ». 

feedback contingencies that the learner needs to progress toward a 
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designafeed competency'^ (James Holland, personal communication).^ The 
extent. to which the learner is exposed to the necessary contingencies is 
reflected in two variables in the instructional events construct (^'quality 
of teaching techniques" arid "quality of materials"). K these variables \? 
show a significant achievement effect, then individualized instruction, 
as defined by the behavioral analyst, is effective. ♦ 

In Gibbons' (1971) descriptive analysis of individualized instruc - 
tion, he, began by indicating that the term "individualized" suggests "a 
s|)ecific instructional procedure adapted to the learning styl^ and person- 
ality of each student" (p. vii). His analysis of the variety of ways in 
which programs and schools attempt to achieve this adaptation to indi- 
vidual students led to the specification of 15 dimensions along which they 
differ. He then plotted profiles for programs (e. g. , Individually Pre- 
scribed Instruction, Winnetka Plan) and schools (e.g., Summerhill, 
Leijcestershire) on these dimensions. Most of his dimensions are highly^ 
related to'one or more of our process variables, and they sample all four 
of our process constructs. Some examples are: "time structure" (pppor- 
.tunity), "pace at which the materials are to be studied" (structure and 
placement), "activity that accompanies or follows study" (motivators), 
and "teaching method" (instructional events). Thus, to determine wheth- 
er or not individualized iia-^truclion, as defined by Gibbons, is effective, 
one would assess the e'ffects-of our proces-s variables that come closest 
to his dimerlisions of individualization. 

Our own definition of individualized instruction is operationalized 
in terms of the program variables, which are reflected by the structure 
and placement, and instructional events constructs. That is, we consider 
classrooms to be individualized to the extent to which they include speci-^ 
ficJ^bjectives, clear mechanisms for matching the student and the curricu- 
lum, and individual sequencing and pacing, and the degree to which the 
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instructional materials and teacher are responsive to individual learner 
needs. To test individualization as we define it would require assess- 
ment of these variables. 

One advantage of the design that we propose is that, as illustrat- 
ed in Section 6, it allows for testing of the effectiveness of a variety of 
definitions of individualized instruction. Such tests are possible because 
we show how to dimension the relevant classroom processes, rathqr 
than just comparing classrooms that fit an arbitrary definition of indi- 
vidualized instruction with classrooms that do not. In addition, we rec- 
ognize that it will be important to identify the kinds of available programs 
that seem to facilitate effective individualization. Therefore, anl^ther 
feature of our design, also illustrated in Sectic^ 6, is that it permits 
the identification of effecti^ existing programs, as well as suggests how 
to improve programs by revealing the most effective mechanisms for 
individualizing instruction, ^ 
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X Section 3 

Assessing Outcomes * * 

The outcome of primary interest in the Individualized Instruction Study- 
will be achievement in reading and mathematics; however, other outcomes 
must also be considered for inclusion in the study. This section describes 
the outcome measure that is recommended for assessing student achieve- 
ment arid the rationale for its selection. In addition, measuremen^t of 
program effects on noncognitive student outcomes and on the total class- 
room environment is discussed. * 

Cognitive Outcomes . 

A The design that we are proposing for the Individualized Instruction 
StlSdy requires early fall and late spring testing. In addition, the number 
of school days between spring testing and fall testing must be stable across 
all participating classrooms in order to ensure the same number of poten- , 
tial days of schooling. It is also essential that test results be comparable. 
In order to stabilize the time of testing, to ensure comparability of re- 
sults, and to minimize the possible effects of frequent retesting, local 
testing should be suspended in the classrooms included in the study and a 
. single pre-selected achievement battery administered. 

To adopt the tests and'testing schedules of participating school 
districts appears to be a sensible approach from the standpoint of saving 
time and rtioney as well as ensuring that local testing programs proceed 
uninterrupted. But in order to have complete and comparable data, the 
contractor who follows this approach would have to patch onto existing 
programs by gathering information on initial abilities or outcomes or 
both. Districts administer tests at a variety of times throughout the en- 
tire academic year, and they use a wide range of homemade and commer- 
cially available instruments. Some have no testing programs at all. 
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Use of an anchoring procedure that wGhald permit the adminis- 
tration of several different batteries was also considered and found- to 
be an unacceptable alternative. To patch together data from a variety 
of achievement test batteries requires a compelling statistical argument 
demonstrating the cemparability of such data. To a certain extent, the 
Anchor Test Study (Bianchini & Loret, 1974) has provided a means where 
by reading scores for the upper elementary grades on each of seven wide 
ly used achievement batteries can be equated. However, tlie results of 
this study are not useful for the present inves^tigation. Data for the An- 
chor Test Study were obtained from and are applicable to children in 
grades 4 through 6. The population that we recommend for the Individ- 
ualized Instruction Study is grades 1 and 3. Also, the Anchor Test Study 
concerned only reading achievement. The proposed effort is aimed at 
assessing achievement iti both reading and mathematics. 

Although a transformation system such as the Anchor Test could 
be incorporated into the present overall design, the expense of such an 
undertaking excludes it from serious consideration. Further, there is 
obviously gr^eater confidence in the comparability o-f outcome data for 
any study when the instriimentation across all groups is as standardized 
as possible. The use of a single measure minimizes concern-about vari- 
ations in test quality, construction, reliability, validity, and adminis- 
tration. 

Another consideration was the possibility of building and using 
criterion-referenced te«?ts i'n the study. The argument for doing so is 
that achievement testing is valid only to the extent that the test is suited 
to a particular Q^rriculum and accurately reflects the curriculum con- 
tent. The argument has force depending upon the use that is being made 
of achievement test results. As diagnostic instruments, for example, 
criterion-referenced tests generally provide more useful information 
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to a teacher than do standardized achievement tests. For the purposes 
of the proposed study, however, this consideration is , not relevant. Giv- 
en the fact that a variety of curricula, individualized and not, will be 
included in the study, one choice would be to construct a criterion-ref- 
erenced test for each curriculxim (an expensive endeavoy requiring ma.ny 
months of effort), or, if the curricula have their own criterion-referenced 
tests, they could be used. This approach, however, would make com- 
parison of results impossible. 

An alternative might be to construct one criterion-referenced 
test for all curricula included in the study. This procedure would also 
be too time consuming. Test development could not begin before May 
1976 and mugt be completed before the beginning of the 1976-77 school 
year. To design, pilot test, revise, produce, and disseminate a test in 
only four months would be impossible. Even if more time were avail- 
able, the outcome measure that would be created -would not necessarily 
be an improvement over, and perhaps not be as good as, standar dize»d 
achievement batteries. For these reasons, criterion-referenced te sting 
was rejected for use in the proposed study and the search for an accept^ 
able standardized achievement battery undertaken. The search was be- 
gun by examining seven widely used achievement test batteries. 

Two of the seven test batteries, the Iowa Tests of BaTsic Skills 
(ITBS) and the Sequential Tests of Educational Progress (STEP), were 
eliminated immediately because they did^ not include forms for the pri- 
mary grades. The remaining five achievement batteries had levels and 
norm data appropriate for grades 1 through 3. 

1. Metropolitan Achievement Test (MAT). 

2. California Achievement Test (CAT). 

3. Stanford AchieveiT\ent Test (SAT). 
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4. Comprehensive Tests of Basic Skills (CTBS). 

5. SRA Achievement Test. 

Detailed reviews were made of the five batteries; these reviews are pre 
sented in Appendix D. 

In evaluating each achievement test battery, several key points 
were considered. Among the most important was content validity. The 
five test batteries considered all make claims to high content validity, 
a claim based chiefly on ho.w the tests were constructed. Tests that 
sampled a wide variety of curricula in use in widely divergent geographi 
areas of the United States and that were approved by curriculum experts 
and teacher educators were, in general, deemed valid tests. ^ 

Further support for the validity of a battery Was found in data 
from analyses of item difficulty and discrimination and from subtest iii- 
tercor relations, and in other data obtained from the development of a . 
standardization edition. Along with validity, tHe reported reliability 
estimates of each test battery were examined. The norm group statis- 
tics and data were also important^ considerations of the reviewers. Ade 
quate sample sizes, acceptable methods of stratification, and efforts 
aimed at eliminating ethnic and racial bias were considered essential 
for an acceptable battery. 

Surprisingly, format quality was quite variable among the five 
leading test batteries. Apart from considerations of the quality oY print, 
illustrations, paper, and layout, the batteries were exarriined to deter- 
mine the degree of sophistication in test-taking skills required of chil- 
dren taking, the tests. Those tests having poor format quality or con- 
fusing layout, or requiring test-taking skills that were considered too 
sophisticated for young children were critically noted. 

The feasibility of using a particular battery was another consider 
ation. That is, it was necessary that thpse batterips being considered 
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have available age -appropriate levels and alternate forms for those 
levels. Where alternate forms were not available, adequate demon- 
stration of the feasibility of using different levels was required (e.g. , 
Level A for assessment of first gra.ders' initial abilities and Leve]^ 
for assessment of their criterion performance). To use different levels, 
correlations between subte sts of these levels must be high. 

A final important focus of the test evaluations was the technical 
manuals that accompany the batteries. These manuals often fail to sub- 
stantiate their claims with data, therefore, full and complete report- 
ing of data from validation studies on^he instrument was deemed essen- 
tial for. attesting to the credibility of the instrument. Time, post, and 
ease of administration were also taken into account and reported; how- 
ever, standardized achievement^test batteries are, in general., compar- 

.able in these areas. * 

* 

On the basis of this evaluation, the Comprehensive Tests of Basic 
, Skills is reconimended for use in the proposed study. (A content analy- 
sis of this battery is presented in Appendix B-1. ) The CTBS stands above 
other batteries as far as validity, reliability, and format /re concerned. 
The development of*the CTBS ha.s been carefully dacumefrited, and all 
claims for the battery are supported by data' reported in its technical 
. manual. This manual is a model document and includes mote informa- 
tion'than is ordinarily found in a publisher's document. In item selection, 
sampling, and norming, special efforts were made by the CTBS develop- 
ers to eliminate test bias against black and Spanish-speaking minority 
groups; Finally, items reflect a balanced sampling from the various 
curricular models current in American schools. Such curriculum sam- 
pling is unavoidable \ti the construction of a national standafdized achieve- 
ment test. Specific items, in fact, do test curriculum-specific learning 
skills (e.g., ^'phonics" versus "look-say" word-attack skills). None- 
theless, the representative sampling from various curricular models 

• 
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serves to place no student at a relative disadvantage simply because of 
his /her participation in a particular curriculum. 

Noncognitive Outcomes and Classroom Environment 

Although the RFP calls for consideration of the "nonachievexnent 
factors which contribute to classroom environment," it doe^ not spell 
out what these factors might be. It was suggested that the designer re- 
view this area and propo'se what definitions and 'instrumentation, if any, 
should be included in the Individualized Instruction Study. Our approach 
to this task has been two-pronged: (1) to determine whether noncognitive 
student outcomes can and should be measured, and (2) to determine whether 
it is possible and desirable to assess the effect of programs on the, total 
classroom environment. 

We do not recommend that noncognitive student outcomes be as- 
sessed in the study for . two reasons. First, although schooling, individ- 
ualized Qr not, may indeed have an effect on some noncognitive-outcomes, 
the theoretical bafeis for such a belief is not well developed. Without a 
sound basis, it is futile to attempt to measure noncognitive or social out- 
comes since it is not cle.ar what to measure or how to make causal argu- 
ments if effects are found. A second argument against the testing of so- 
cial outcomes is that their measurement in the primary grades is still 
in a primitive state. 

> Our consideration of noncognitive or social outcomes began with 
the generation of a list of outcomes that designers of instructional pro- 
grams have claimed will be affected by their program^ , self-con- 

cept, inquiry skills, autonomy). The jiext step was to locate instruments 
that purport to measure these specific outcomes. 'The short duration of 
the ^tudy ruled out the possibility of developing such instruments from 
scratch. Existing instruments werQ located, screened, and eliminated 
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from further' consideration if they failed to meet any one of the follow- 
ing criteria: '* 

1. The instrument could not be highly correlated with 
reading and mathematics ability. If it were, it 
would measure little not already measured by the 
achievement test battery. 

2. The instrument had to measure the social variables' 

r 

* in question, i. e. , it had to be valid as measured by 

standard measures of validity. 

' ' •■ \ ^ 

3. The instrument had to be reliable as measured by 
standard measures of reliability. 

4. T*he instrument must have been designed or adapted 
for use in the primary grades. 

-J 

5. The instrument must be usable from an administra- 
tive standpoint. This criterion would rulq out instru- , 
ments that are described in the literature but arp 
otherwise untraceable, those that require an exorbi- 
tant amount of pupil/examiner time (in excess of 

three hours per pupil), /nd those that require a highly 
trained examiner ot coder. A number of projective 
tests like doll-play were eliminated under this cri- 
terion. 

The results of the search for an instrument that would meet the^ 
criteria were disappointing. Nat one instrument of the many ^considered 
was totally acceptable. Table 3. 1 lists some of the tests tha>^ were re- 
jected and a criterion they failed. They may have failed other criteria, 
but this information was not recorded because the test reviewers elimin 
ated an instrument uppn failure to meet one criterion. 
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Table 3. 1 



Instruments Co'tisidered and Criterion They Bailed 



* 

Instrument 


Criterion Failed 
r 


Early School Personality Questionnaire 


2 




Carel Instruments, Form S-B 


3 




Edwards Personality Inventory 


4 




The Thomas Self- Concept Values Test 


3 




California Test , of Personality, Primary Form AA 


3 




r 

How I See Myself, Elementary Form 


4 




Problem Expression Scale 


4 




Sears Self-Conceot Inventory 


2 




Attitude Toward School, School Sentiment Index, Primary 3 




Coopersmith Self-Esteem 


4 




Piers-Harris Children's Self-Concept 


4 




Driscoll Playkit 


5 




Animal Crackers 


^ 2 





/ 



Numbers refer to the criteria listed on the previous page. 
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One of the tests considered, though it failed the criterion regard- 
ing content validity, did include several worthwhile features. This test, 
*'Animal Crackers, " is reviewed in, detail in Appendix D, but mention 
will be made here of several of its characteristics. First, in its five-, 
scales, it covers a range of social outcome'g'-ftQ^ Purposiveness to 
School Enjoyment and, 'therefore, casts a sufficiently widiiMi'et as to pick 
up any positive or unintended negative effects of school. Second, the test 
items, in the main, address school- centered behaviors and attitudes so 
that they are likely to tap effects related to school rather than home or 
community effects. Third, the instrument can be administered in both 
the fall and spring, thereby making statistical control possible. Finally, 
the testes predecessor, Gumpgookies, has been widely used in previous 
evaluations of Compensatory education programs. Although we are not - 
recommending that Animal Crackers be included in the study, the con- 
tractor may be interested in reviewing it to get a sense of the limitations 
^of even the '"best" of the currently available measures of noncognitive stu 
dent outcomes. 

Our recommendation regarding assessment of the effects of 
schooling on the total classroom environment is that no attempt be made 
to measure these effects. There are two reasons for this recommenda- 
tion. First, what these effects might be is not clear,- so it would not be 
possible to construct a priori measuring instruments* to tap them. In- 
stead, these effects would have to be determined in situ. To do so would 
require one highly trained observer /ethnographer per instructional model 
These individuals would have to identify, describe, and then measure the 
effects of schooling on the environment. J^urther, their efforts would 
have to be very well coordinated to ensure comparability of results. The 
tfost of such an endeavor would b^ prohibitive, even in $1. 5 million study, 

A- second aad more compelling reason for not studying the unspec- 
ified environmental effects of schooling is logical rather than logistical. 



Instructional programs are designed to impact academic achievement. 

, Changing the classroom environment is the, means to accomplish this 
end. We cannot treat environm'biatkl effects as both independent vari- 
ables (means) and dependent variables '(^nds). We must assume that 
if a given instructional strategy has any impact, on the classroom envi^ 
ronment, then this impact ^11 be reflected in the cognitive outcome 

. measures . * ' 

We do recognize, however!, 'tHb*'value can be attached to both 
means and ends. Everyone "k-nows" that the Vad^gljdo not justify the m,eans. 
In fact, most parents are as concerned about the kinds of experiences their 
children have in school as thfey are about wKat their children learn. We 
\ believe that the way to respond to this need is to use our process vari- 
^^bles as descriptors of what classrooms are like and as predictors of 
how much is learned by children in those classroom's. 
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Section 4 

Sampling and Securing the Cooperation of Schools 

* 

Sampling ' ■ ' 

Based on both statistical and cost considerations, it is recom- 
mended that 400 classrooms .be included in the Individualized Instruction 
Study. This sample size is large enough to enable the contractor to in- 
clude 16 variables (initial abilities, criterion perfoVmanoe, and the 14 
progifam and support vaifiables) in the data analyses. >It is small enough 
to rriake intensive examination of classroom processes affordable in a 
$1.5 million study. 

Further, ifis recpnomended that 200 of these classrooms be at 
the first-grade level and 200 at the' third-grade level. Grade 1 is rec- 
ommended because it is very irr;'f)ortant to assess the effects of educa- 
tional programs as early as possible. Not only is grade 1 the most cri- 
tical ye^ri for beginning reading- -a most important and highly valued 
skill, but it is also the first grade level at which reasonable confidence 
can be placed in test results. In addition, it is the earliest grade for 
which there ife reasonable consensus as to the objectives of the reading 
and mathematics programs. 

Given that grade 1 was to be included, the choice ^a^s then whether 
to include other grade levels and, if so, whic1l*bne(s). It did not seem 
necessary to include every grade suggested for consideration^in the RFPJ 
i.e., kindergarten through grade 4. Studying more classrooms, at a few 
grade levels seemed more desirable than studying several levels with 
fewer clasSsrobms per tevel. Including at least one other gr^tde level in 
addition to grade 1 was deemed valuable because it is possible that effec- 
tive classroom processes may be different at different grade levels. 

Grade 3 was selected as the other grade level to study. It was 
chosen over grade 2 because it was felt that there is considerable 
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"^ajdvantage in spreading the information over the age ranges (grade levels) 
of interest. Further, some individualization processes become more 
fully operational at grade 3, where the need for intensive small-group » 
instruction is diminished. Also, in grade 3, achi-evemerlt expectations . 
are different than they are in ttte earlier grades . Reading tests, for 
example, change from the first to the third grade, malting it important 
to examine changes in effective classroom pjractices as desired reading 
outcomes change. 

Grade 3 was preferred to grade 4 for two reasons. First, by 
grade 4 r^ny programs require that children^ove from classroom to 
classroom or teacher to teacher for different subjects. As a result, it 
is difficult to assess the relative independent effects 'of classroom proc- 
esses for given children. Second, the entering abilities of higher -grade 
students become increasingly confounded with previq\|s s chootng, thus 
overwhelming the potential for explanation of end-of-ye^P^TWmrmance 
by current processes. 

Kindergarten was not selected for investigation because the initial 
ability testing of kindergarten children is extremely difficult, particularly 
since group testing is impossible. In addition, the vastly different ex- 
pectations as to what is to be accomplished in ^kindergarten make it diffi- 
cult to arrive at a ^nsible measure of kindergarten outcomes. 

In defining a'^planfor selecting specific classrooms, there were 
several major considerations: 

1. The primarx^ways in which classroom processes can 
vary must be i^presented in the classrooms selected, 
since the effects of classroom process on achievement 
cannot be established unless there is adequate variation 
in pi?ocess. 
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2. The number of curricula in use in the sample class- ^ 
• rooms must be kept to a manageabljje size so" that it 

will be possible to: (a) estimate the- degree of corr-e- 

spondence between objectives covered ^ the curricu- 

lum and objectives sampled by the outconie measures, 

♦ * 

and (b) estimate the instructional efficiency of the m^lt-^ 
terials used. These analyses would be very expensive 
and time consuming if a different curriculum were 
operating in each participating classroonl. ' 

3. Geographic representativeness is not essential, 
since tKe Individualized Instruction Study is not 
aimed at estimating national parameters such as the 
frequency with which various forms of individualized 
instruction are used in compensatory educat^ion. 

• 

4. The classrooms to be selected should inclucfe children 
eligible for compensatory education programs. How- ^ 
ever, it is not necessary or even desirable to restrict 

the sample to compensatory programs. Low achieving 
children will be present in almost any classroom. The 
'study does not require representativeness of present 
compensatory programs, but it does require repre- 
sentation from a variety of approaches to classroom 
instruction. 

Lists of specific schools that would participate in the Individual- 
ized Instruction Study are not being provided at this time for two reasons: 
(1) the identity of the^ stucjy contractor* is unknown, and it is recommended 
that participating schools be located in the immediate vicinity of the con- 
tractor's headquarters; and. (2) it did riot seem appropriate to seek school 
cooperation for a study that is only in the" design stage. What is provided 
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in this section is a set of procedures for identifying schools that could 
be asked to participate in the study and {or securing their cooperation. 
It should be noted at this point that sampling for this type of study is 
necessarily an interactive process, and that continual adjustments must 
be made between what is desirable and what is possible. 

The first step in identifying possible schools is to determine 
which two or three states in the immediate vicinity of the study Con- 
tractor (or data collection subcontractor) to include. To do this, the 

/ 

states that show adequate variation in program and type of poverty (e. g. , 
urban,/ small town, rural) must be identified. If, for example, the study 
contractor were located in Pennsylvania, one might begin by consider- 
ing Pennsylvania (two major cities and several smaller cities, small 
town and rural settings) and West Virginia (small ^ties and towns and 
rural settings including parts of Appalachia). If upon further examina- 
tion, ^n adequate variety of programs cannot be identified in these 
twQ: states, then Ohio, for example, could be added. Even if one state 
shows adequate vari^tiqn, at least two states should be sampled so as 
to make the study results more convincing. 

The next step is to identify school districts in the states selected 
that appear to be operating individualized programs. To do so, the fol- 
lowing lists should be obtained: (1) schbols participating in the national 
Follow Through program; (2) schools implementing one of the nationally 
available individualized programs (e.g.. Individually Prescribed Instruc- 
tion, Individually Guided Education, Project PLAN); and (3) schools par- 
ticipating in District Survey I. These lists and descriptions of the^ 



Students who will participate in the instructional setting contrast de- 
scribed in Section 6 will also be selected frorii District Survey I "schools. 
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programs are available^rom the United States Office oi Education, com- . 
mercial publishers, and NIE, respectively. From the lists and descrip- 
*tions, it will be possible to determine the number of school districts that 
could participate in the study, the features of individualization that appear 
to be incorporated in the programs ^.being implemented in those districts, 
and the types of programs in operation (i.e., reading and/or mathema- 
tics). Programs descriptions often do not reflect the exact nature of tha * 
program. However, a general notion of the program's characteristics, 
can be obtained from this source. ? 

Table 4. 1 illustrates, the results of our review of program par- 
ticipants and descriptions for Pennsylvania, West Virginia, and Ohio. 
The programs listed in this table are all operating in Pennsylvania and/ 
or Ohio. The information 'that we reviewed indicated that only one in- 
dividualized program is operating in West Virginia, so that state was ex- 
cluded as a p(!^ssible study site. Eight programs were identified that in- 
corporate at least one o£five features of individualization that can be 
estimated from program descriptions: . (1) stkted performar\ce objectives 
that are cognitively oriented, (2) systematic diagnosis of the instructional 
needs of indijWdual children and provision of instruction based on that di- 
agnosis, (3) systematic monitoring procedures to assess mastery at regu- 
'lar intervals, (4) procedures for the adjustment of instruction with respect 
to rate and/or content based on systematic assessment of student progress, 
and (5) specific- instructional materials that have been developed or iden- 
tified for use in individualized instruction. These five individualizing fea- 
tures are all associated with the structure and placenient characteristics 
of a program. Since ensuring variation in classroom processes is the 
primary objective of the sampling plan and since these features represent 
the major ways in which individualized programs vary, it is fortunate that 
adequate variation in the five features can be established early in the sam- 
pling process. ' The other constructs in the classroom processes model 
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that we are recommending (i. e. , opportunity, motivators, and instruc- 
tional events) cannot be estimated from program descriptions. This is 
not a problem, however, since our experience has shown that variation 
on these three constructs can be found in any set of ZOO classrooms. 

As Table 4. 1 indicates, there are at least 90 school dis\ricts-in 
Pennsylvania a^nd Ohio that could be asked to participate in the study. 
Ninety districts will not be needed, but not all districts will agree to par- 
ticipate. Replacement can be -achieved by selecting alternative districts 
that are operating similar instructional programs and that maintain vari? 
ation in type of setting (urban, small town, rural). We recognize that 
there may be attrition of schools during the course of the study. How- 
ever, the sample size that we are recommending will permit up to 20 
percfent attrition. 

Once identified,, candidate school districts should be contacted 
to determine: (1) their willingness to participate in the study; (2) the 
number of fitst- and third-grade classrooms in the district that are op- 
erating an individualized program in reading and/or math; (3) the indi- 
vidualizing features of that program; and (4) the number and type of stan- 
dardized programs operating in grades 1 and 3 in the district (i.e., pro- 
grams exhibiting none of the individualized features listed earlier). From 
.the school districts that are willing to cooperate in the study, 200 first- 
grade and 200 third-grade classrooms must be selected. ^Table 4.2 il- 
lustrates the procedure for sampling first-grade classrooms within a 
school district. The general recommendation is to select up to 20 elass- 
roorris per grade level per district, about half of which are operating in- 
dividualized programs, and to keep the number of districts to under 20. 
In selecting individualized classrooms, preference should be given to 
those classrooms that exhibit a unique combination of individualizing fea- 
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and, that are located in districts where standardized classrooms are 
available for comparison purposes. Also, there, must be adequate rep- 
resentation of type of setting. At least 50 percent of the classrooms 
should be located in urban settings. 

* 

After specific individualized classrooms are identified, school 
achievement test means should be obtained for the schools that include 
these classrooms, as well as for schools, in the district that are oper- 
ating standardized programs. Standardized classrooms should be se- 
lected from schools with test means as similar as possible to those of 
the individuaKzed classrooms. 

' Securing the Cooperation of Schools 

Once the states that will be included in the* study have been iden- 
tified, the Project -Director and/or School Coordinator and, if possible, 
a representative of NIE should personally contact the chief state school 
officers to enlist their cooperation.'* Meetings are a far more effective 
means of obtaining cooperation than are written or telephone requests. 
'It should be emphasized that the purpose of the study is not to evaluate 
schools or classrooms, but rather to identify effective mechanisms for 
improving instruction. At the local level, initial contacts should be 
made with district superintendents and school principals by the Project 
Director and/or Sghool Coordinator. In addition, there are a number 
of incentives that would increase the probability of a district agreeing 
to participate in the Individualized Instruction Study, as well as ensure 
cooperation throughout the course of the study. Not all sqhool person- 
nel will be interested in the same incentive; therefore, it will be impor- 
tant to match incentive to the individuals involved. Some possible incen- 
tives are as follows: • , ... 
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1. Make available data summaries ponsidered to be im- 
portant by'fhe jiistrict. This information would be 
available within the data collected for the study. 

2. 'Offer to share with the cooperating district the study 
results as soon as possible. Delayed reports are not 
well received. 

3. Make training sessions required for school personnel 
as attractive as possible, e. g. , schedule at their con- 
venience, compensate for time if sessions are not 
held during working hours, 'conduct in comfortable 
facilities. 

4. Provide copies of previously prepared research re- 
ports or papers that arei relevant to the study being 
coj!iducted, . ^ 

5. Extend invitations to attend, as guests of the con- 
tractor, a symposium that would. focus on the nature, 
purpose, and importance of the study, and would in- 
clude participation by NIE representatives. 

Other possible incentives may be identified in discussions with school 
personnel. 

Once schools have agreed to participate, the next step is to ob- 
tain the^cooperation of parrticipating and nonparticipating teachers in 
each school. Both groups of teachers should b^ informed of study actiyji- 
ties. In addition, since extra demands will be j>laced on participating 
teachers by the testing, interviews, and videotaping in their classroom, 
it^s suggested that they be compensated in some way. Our recommenda 
tion is that teachers be paid an honorarium of $50 each upon completion 
of the spring testing. Doing so will cost $20, ODO, but it could mean the 
difference between a successful or unsuccessful $1, 5 million effort. 




Section 5 * 

Collecting Data 
4 ^ * 

In the study that we are prop^js^ng, data will be collected through 
in- class, interviewing of teachers, analyzing curricula, videotaping class- 
room activities, afid administering the Comprehensive Tests of Basic 
Skills. This section presents recommendations regarding the scheduling 
of all data collection activities, the personnel required, the training of 
th,ese personnel, and procedures foi: monitorfeg the quality o^ the data 
collection effort. 

ln-0ass Ini'^r vie wing 

In-class interviewing of teachers will provide data on three of the 
process constructs: structure and placement, opportunity, and motiva- 
tors. The In-Class Teacher Interview Questionnaire is included in Ap- 
pendix A, along, with- directions for obtaining and coding the i^iformation 
required. Interviewing will take place twice during the school year: 
October-November •1.976 and March 1977. Site Coordinators, under the 
supervision of the Field Coordinator, will be responsible for scheduling 
interviews with school personnel at each site. (The positions of Site 
Coordinator and Field Coordinator are discussed in Section 7, along 'with 
other key project positions. ) 

J 

. Ideally, the interviewers should be persons who have a good idea 
of what goes on in the classroom, e.g. , former teachers. The Fi61d 
Coordinator is responsible for the training of all interviewers. He/she 
should conduct one or two training sessions to review the questionnaire 
and the directions for obtaining and coding the information. It m^y be 
beneficial for the interviewers to practice administering- the questionnaire 
•several times to each other. If possible, arrangements should be made 
for the interviewers to administer the quef stionnaire to teachers in schools 
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that are not participating in the study prior to interviewing participating 
teachers. It should be made clear to the interviewers that they should 
probe for accurate and complete information rather than adopting the 
traditional survey approach of trying to fit the first response given into 
a predefined category. 

9 

An interview will take approximately 1. 5 hours per classroom 
to complete. That is, to collect data from 400 teachers will require 
600 hours in the fall -and in the spring. Using the same personnel to 
conduct both interviews will be an advantage in terms of ease of train- 
ing and confidence in the reliability of the information obtained. 

Coders of the interview data should be college students or clerks. 
Depending on contractor preference, the interviewers may also serve 
as coders. The Field Coordinator's responsible for thd draining of all 
code^rs. The coordinator (and perhaps the interviewers) should explain 
the coding of each question to the coders in the falL In addition, it may 
be useful for him/her to meet with them in the spring to explain the var- 
ious methods for combining fall and spring codes into a single code. Any 
questionnaire answers that. are unclear to the coder should be clarified 
with the interviewer who completed the questionnaire. Coding a single 
questionnaire requires approximately one hour. That is, to code ques- 
tionnaires from 400 classrooms will require 400 hours in the fall and 
in the spring. Using the same personnel for coding both fall and spring 
data is recommended. 

. As the interviews are coded, they should be grouped into batches 
of ten. In order to maintain high levels of quality control, one out of 
each batch of interviews should be recoded. If errors are discovered, 
the entire batch of interviews coded by the particular individual making 
the detected errors should be recoded. -All coders should initial their 
work so" as to permit the identification of individuals who make consistent 
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errors and, if necessary^ the reassignment of tasks. In addition, it is 
recommended that informal lists of item responses be compiled by batch 
so that obvious errors can be detected before the data source and th^ 
data become too separated. 

Analyzing Curricula 

Information on all four of the process constructs will be provided 
through an analysis by curriculum experts of the curripula in use in par- 
ticipating classrooms. A Curriculum Analysis Questionnaire is included 
in Appendix B-1, along with directions for obtaining and coding the infor- 
mation required. Additional information will be gathered through a be- 
havioral analysis of each curriculum. Instrumentation and directions 
for conducting such an analysis are presented in Appendix B-2. ^ 

Preparations for curriculum analysis must begin before the end 
of the 1975-76 school year. That is, in May 1976 a list must be com- 
piled of the reading. and mathematics curricula that are expected to be 
used in each classroom so that the materials needed by the curriculum ex- 
perts can be ordered. In the fall, a final list of curricula should be pre- 
pared and any materials ordered that were not on the previous list. It 
is recommended that Curriculum analysis be initiated in September 1976 
and completed in December 1976. ✓ 

The curriculum experts should have experience in instructional ' 
design. Training could take one to three weeks, depending on its iiiten- * - 
siveness and the background of the experts. It is suggested that the Field 
Coordinator, who is responsible for training, become familiar with a text 
entitled The Analysis o f Behavior in Planning Instruction (Holland, Solomon, 
Doran, h Frezza, in press). This text is in the form of* self-in'structional 
units, several of which might serve as the basis for training on the be- 
havioral analysis of a curriculum (e.g.. Units 18, 19, 20, 28, 31). These 
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units could be supplemented with a minimum of didactic material and 
a great deal of elementary math and reading lesson samples on which 
the curriculum experts would be asked to count response* opportunities, 
identify response contingencies, and estimate. potential error rate. Be- 
havioral analysis training could also be provided by a number of indi- 
viduals or their students (e.g. , Eva Baker, Center for the Study of 
Evaluation; James Holland, University of Pittsburgh; Susan Markle, 
University of Illinois at Chicago Circle). 

The more familiarity the curriculum expert has with the cur- 
ricula used in participating classrooms, the more rapidly he/she will 
be able to evaluate them. It is estimated that an expert will require 10 
to 20 hours to complete and code the Curriculum Analysis Questionnaire 
for each classroom for each curriculum (i.e., for reading and for math- 
ematics). An expert who is well acquainted with the curriculum will ^ 
probably require 10 hours, whereas an expert who has never seen thev 
curriculum before may require as much as 20 hours. A behavioral anal- 
ysis of the 'curriculum will require 2 to 4 ^hours when all children in a 
classroom have identical assignments. When all children have different 
assignments, the analysis may take 10 to 20 hours. 

Quality control procedures similar to thosu recommended for 
the feoding of interview data should be followed. However, because of 
the grBater amount of time required to analyze a curriculum, onl^^^^ne 
of every twenty-five analyses should be redone. If errors are discovered, 
five more analyses should be reexamined. If additional errors are found, 
the entire batch should be analyzed again. 



Videotaping 

Although videotaping of in-classroom activities will yield infor- 
mation on or confirm previously collected data on all four of the process 
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Qonstructs, -its main purpose is to clarify tKie instructional events in a 
classroom. Videotaping will occur both in the' fall (October -November 
1976) and. in the spring (March 1977). In addition, gome videotaping 
should be done in nonparticipating classrooms before the end of the 1975- 
76 school year to provide practice for the videotaping team as well as 
tapes that can be used in training sessions for the observers. The Site 
Coordinators will be responsible for scheduling videotaping at each, site. 

The collection of data through videotapes requires videotapers, 
assistants, and observers. The tapers should be knowfedgeable in the 
use and repair of audio -visual equipment. The assistants should have 
experience working in schools as teachers or field resea,rcher s. Video- 
tape observers or coders should be exper^ienced in in-classroom observa- 
tion. It is strongly recommended that the feame observers, as well as 
videotapers and assistants, participate in data collection in both the fall 
and spring. To tape one classroom will require 4 hours in the fall and 
in the spring (i. e. , 2 hours for reading and 2 hours for mathematics). 
Coding a videotape from one classroom will take 3 hours in the fall and 
in the spring, or^a total of 6 hours during the school year. 

In Appendix C are directions for producing videotapes, for ob- 
taining measures from videotapes, for coding videotapes, and for main- 
taining quality control. The information presented is sufficiently detailed 
to provide the basis for training sessions to be conducted under the super- 
vision of the Field Coordinator. 

Administering the CTBS 

As indicated in Section 3, the CTBS is recommended for use in 
measuring students' initial abilities and end~of~year achievement. The 
proposed testing schedule is as follows: 
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September Pretest May Posttest 

Grade 1 CTBS-Level A CTBS-Level B 

Form S FormS 

Grade 3 CTBS-Level 1 • CTBS-Level 1 

Form S S Form T 

It is suggested that! two subtests of the grade 1 battery aot be administered. 
These subtests are "Language" in Level A and "Language II" in Level B. 

Although using classroom teachers as testers is admittedly the 
most convenient and inexpensive way to implement the testing program, 
this approach also has the most .potential for introducing bias into^a test- 
ing program. For this reason, it is suggested that participating teachers 
not administer -the test batteries to their class. Instead, outside testers 
should be hired, (^ne possible source of testing personnel is the pool of 
available substitute and/or retired teachers in a locality. The Site Co- 
ordinators, under the supervision of the Testing Coordinator, should con- 
tact local school boards participating in the ^tudy and request a list of 
substitute and/or retired teachers who are available. (The position of 
Testing Coordinator is described in Section 70 Test administrators can 
then be selected from this li^t. 

In addition to having nonbiased testers, it is important that all 
test administrators be thoroughly trained in a standardized testing pro- 
cedure. It is highly recommended that all testers be trained by the same 
individuals. To ensure that all -testers will follow the same model, a filrxt 
or videotape should also be produced to demonstrate model testing be- 
havior. 

An adequate trairjing program and competent supervision of the 
test administrators by the Testing Coordinator and Site Coordinator s% 
should se^ve to establish a high level of quality control in test adminis- 
tration. As an added check, the Testing Coordinator may ask the Site 
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Coordinators to monitor testing procedures in a sample of classrooms. 
A checklist- of administration could be drawn up based on the standardized 
procedure set forth in the training sessions. Any variation in this pro- 
cedure would be noted and corrected by the Site Coordinators. However, 
given the expense of such a procedure and the type of training and super- 
vision proposed, this extra check of te^t administration should be imple- 
mented at the contractor's discretion. 

Another quality control procedure, one that^is strongly recom- 
mended, is initial verification of the tests before they are o^eturned to 
the Testing Coordinator. Answer booklets without names, stray- marks, , 
missing data, etc., should all be corrected before students are dismissed 
and the testing considered complete. The testers should be responsible 
for this phase of quality control. 

Just as important as the quality of test administration is the qual- 
ity of test scoring. At the present time, considerable detjate surrounds 
the issue of hand versus machine scoring. Our experience, as well as 
the experience of other organizations such as the Stanford Research In- 
stitute, has indicated that there is' considerable savings in cost, as well 
as greater accuracy, with hand scoring. The superiority of hand scoring 
has been found to be especially true in the scoring of younger children's 
protocols, grades 1 to 3, where answer sheets cannot be used. Since 
every test must be scored at least twice in order to validate scores, it 
is recommended that an initial hand scoring followed by a second hand- 
scored validation be adopted as the most economical and accurate pro- 
cedure. K ^ , * 

Workshops * . 

It is suggested that a workshop be held for data collection person- 
nel in July 1976 to familiarize them with all aspects of the field work. 
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A further recommendation is that district 'supervisors, principals, a 
perhaps Site Coordinators be invited to a workshop at th.e contractor' 
office to acquaint them with the purpose, scope, and time demands o 
the study. This workshop should take place ip. August 1976. 
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Section 6 
Reducing and^ Analyzing Data 

This section summarizes the procedures that we recommend for 
reducing classroom process measures to a small number of variables, 
as outlined in Section 2, and for analyzing the resulting variables to re- 
veal the effectiveness of various classroom processes ifii the teaching of 
reading and mathematics. In addition, a small-scale study is described 
that should be conducted parallel to the main investigation of classroom 
processes. This special study contrasts .in-classroom individualization 
strategies with the practice of "pulling" disadvantaged children out of 
classrooms for special tutoring, a frequently used Title I program ap- 
proach. 

Reducing Data 

To describe and analyze the large amount of information that will- 
be collected in the Individualized- Instruction Study requires that the class- 
room data be systematically reduced to a manageable number of process 
dimensions. It is conceptually impossible and statistically undesirable 
to analyze 200 measures'separately, for exaipiple. Fortunately, the meas 
ures described earlier were constructed with this consideration in mind. 
The basic approach that we propose is to reduce the measures along th'e 
lines of the six constructs of the classroom proces s 'model illustrated in 
Figure 2. 1. The steps involved in data reduction are: 'elimination of 
unusable measures, preliminary correlation and partial correlation analy- 
sis by variables withi'n constructs, inspection and reflection of measures, 
plotting and transformation of data, development of standard-lcbre s with 
unit variance, and combination of measures to form variables. By com- 
bining the data in a manner that preserves the initial "meaning" of the 
numbers, the final results will be more interpretable . Also, unpacking 
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of the data for more detailed analysis .of any particular construct can be 
carried out in a straightforward manner. 

Elimination of measures . In any large-scale data collection 

effort, some of the information will be unusable for one reason or an- 
> 

other. The purpose of this first step is to eliminate such measures. 
Means, standard deviations, and skewness should be examined, and meas- 
ures with zero or near zero variance should be deleted. Measures wi 
unusual skewness should be examined for outliers to check for obvious 
errors (e.g., if inspection reveals one classroom with 300 students, a 
check should be made since" t\iat value is highly unlikely for that meas- 
ure). Another way of identifying outliers is to compute frequency dis- 
tributions and scan them for ''unusual'^ values. Obviously, not all un- 
usual values are incorrect, but simple clerical errors can be detected 
in this way. 

Correlation and partial correlation . The next step in data re- 
duction is to inspect the correlations among the measures for each of 
the variables. Correlations should be computed on the measures along 
with data from initial and end-of-year student performance aggregated 

as classroom means. These correlations should be inspected, particu- 

% 

larly^with respect to unsually high relationships between initial abilities 
and process measures. Identifying such relationships will be helpful in 
the later interpretation of commonalities, should they exist. Also, the 
signs of th^ relationships between measures and outcome residuals need 
to be inspected. This task can be accomplished by partial, cor relation, 
removing initial ability^from process and outcome, ^or by correlating 
process with achievement gain. The objective is to identify measures 
that do not conform to theory ^^th respect to their effect on achievement. 
This procedure is a conservative form of criterion scaling, which at least- 
ensures tl^iat a measure does not need to be reflected before it is combined 
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with other measures. A negative partial would indicate the need to con- 
sider reflecting a measure. 

Data transformation . One of the assumptions of the analytic ap- 
proach that we are advocating is that each process measure relates to 
the outcome measure in a linear fashion. It is possible, however, that 
some of the measures relate in a nonlinear way. The purpose of this 
step is to determine whether any measures need to be transfornjed so 
that the , -eventual combination of measures into variables does not result 
in loss of information. Procedures for determining whether a measure 
needs to be transformed and for selecting appropriate transformations 
have been summarized by Tukey (1970) and need not*^be detailed here. 
After the measures are transformed to make them additive, they need 
to be standardized with a zero mean and unit variance prior to combining 
them. . \ 

Combination of measures. The recommended procedure for 
combining measures into variables is to simply add the measures for 
each variable after they ha,ve been adjusted to unit variance, with sep- 
afate scalings for math process and reading process. This procedure 
reduces the classroom data from over 75 measures down to a manage- 
able number of variables. These variables should be combined Vith ini- 
tial abilities and outcomes for data analysis. Separate analyses should 
be conducted for reading and for mathematics, and for g.rade 1 and grade 
3. ' 

* 

Principal components analysis was one of the alternatives con- 
sidered for combining measures. The first principal component repre- 
sents a most reliable single dimension of what is common to that set of 
measures. However, it does not incorporate variance in measures that 
are unrelated to the set. For exa^mple, obtaining a single dimension of 
socioeconomic status from a set of correlated indicators such as income, 
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occupation, and education can be accomplished by using the first prin- 
cipal component for that set. If a measure in that set does not correlate 
with other measures in'the set, it will not load pr^^the first component. 
This is not serious if one is attempting to scale a construct like socio- 
economic status, which is viewed as a single dimension that "explains" 
the relationships among the measures and/or that represents the vari- 
ance common to the set. 

Measure^ of a particular classroom variable are not necessarily 
correlated with each other. Thus, in combining classroom measures, 
what determines whether a measure should be combined with others 'ycy 
the set is whether or not the measure has face validity for the varijable 
being g,ssessed. In Table 2.2, for example, there are 10 measures to 
^be combined into a "quality of teaching techniques" variable. Teachers 
high on one measure, will not necessarily be high on others. If, however, 
one assumes that the more of those behaviors present in the classrpom, 
the more effective the teaching, then a sum across all such measures 
would be an indication of "quality of teaching techniques. V 

Analyzing Data 

. ' * 

The RFP requires a plan "for arguing causal relationships be- 
tween program and outcome variables" in a nonmanipulatory survey of 
existing classroom practices, despite the difficulty of such arguments 
under the canons of statistics (as reviewed by Cronbach & Furby, 1970). 
Since everyone knows that correlation cannot prove causality, what is 
required are analyses that create the strongest valid presumption of 
causality. Although there are analyses of correlations that permit valid 
and, stronger presumptions of causality under ideal research conditions 
(e. g. , path analysis^), the actual conditions for the Individualized Instruc- 
tion Study call for a conservative method of analysis. On the technical 
side, this. is. because the measurements employed will possess only partial 
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validities for the constructs they represent, and will have variable 
and unknowable degrees of unreliability. On the policy side, this is 
because there is less hazard to the nation when r<e sear cli under estimates 
for Congress the impact of parj^icular arrangements' for education than 
when research claims unreplicable effects. Commonality analysis pro- 
vides an appropriately conservative approach to arguing causality in the 
results of the study. , ^ v 

' Commonality analysis is a comparison of statistical models for 
the data. When it creates a presumption of causal efficacy for program 
variables, it does so by reducing the probabilities for alternative explan- 
ations of the observed educational development of pupils. The basic fact 
about survey research data is that the possible causes* of outcomes as 
described by alternative models are confounded to some extent. Rather 
than attempt hazardous uriconfounding by algebraic tricks, comrfiqnality 
analysis represents the confounded portions of effects on outcomes as 
separate partitions of the outcome variance (called "commonalities"). 
These confounded portions are not included in estimation of the irreduc- 
ible effects of the separate possible causes (called their "uniquenesses"). 
The conservativeness of the 'method resides in this use of minimum rather 
than maximum estimates of separate effects. The advantage to the method 
rests with the visibility of the commonalities; other conservative approaches 

tend to mask them. ' . 

■ ' • 

When the uniqueness for program variables has b^kn computed 
from the data as a result *of comparisons of modela using the combina- 
tions of initial abilities, support, and program variables,, the precise 
interpretation of it-wilj be that the uniqueness of the program variables 
is the portion of the variance in educational development that can only be 
explained by employing the measurements of program* variables in a com- 
plex model for the outcomes. No. model employing only the initial abili- 
ties and support variables can account for this portion of the variance 
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in educational development. Granting that this does not prove the de- 
gree of causality of the program variables, to the extent to which the 
models under comparison are judged complete and competent it justi- 
fi/ss a presumption of degree of causality. 

While the external validity of a survey study is dependent on the 
characteristics of the samples achieved, the internal validity is depend- 
ent on the qualities of the models compared. Causal arguments will be 
most convincing in th^- presence of complete and technically competent 
contrasted models^ The omission of probable caujses that are known to 
be correlated with the hypothesized cause of central interest is devas- 
tating. Unconvincing operationalizations of constructs or ridiculous un- 
reliabilities are debilitating. Exhaus*tiveness and redundancy are the 
rules to follow in making observations. Parsimony and organization 
are achieved by combining detailed measures prior to, commonality anal- 
ysis. That is, constructs for the initial abilities, support, and pro- 
gram variables are scaled as linear functions of exhaustive redundant 
observation scales. Th,e gain in reliability for a positive linear function 
of a set of positively correlated indicators is substantial and should not 
be overlooked. 

The model that has guided our design of the Individualiaed Instruc 
tion Study postulates that a particdiier set of program variables describ- 
ing ways and degrees in which instruction is individualized is essentials 
to the explanation of pupil learning in reading and mathematics, that oc- 
curs over the study year. The model holds that this set of program vari- 
able3 combines with two other sets of variables, the initial abilities and 
support variables, to generate pupil learning. The alternative models 
that have to be shown to be inadequate in the data analysis hold that: (1) 
initial abilities are sufficient explanation of pupil learning, and (2) abil- 
ities together with support variables are sufficient explanation of learr 
ing. Since these are simpler explanations, their truth probabilities can 
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only be reduced by showing appreciably greater power for the complex 
raodel involving the addition of information^ i^bout program variables. 

Table 6. 1 illustrates the most general result from a sequence 
of possible commonality analy^s, in which the total variance explained 
{i.e., the squared multiple correlation) is . 6'8, using gain in- mathe- 
matics achievement as the dependent v'atriabla and 1 5 abilities and proc- 
ess variables as the predictors. (These variables are listed in Table 
6. 5.) Table- 6. 1 indicates that 18 percent of the variance in improved 
mathematics performance can be attributed to classroom processes. 
It also shows that 22 percent is due to process and/or ability, and is 
the result of correlations between process measures and initial ability 
that occurred among participating classrooms. This -commonality is 

the extent of the in-commori effect for process -and ability that cannot 
♦ 

be sorted out in this particular study. It is partly* a function of the fact 
that better classroom processes tend to be available to more able stu- 
dents. One objective of the sampling design is to plan classroom proc- " 
ess variation so that it is as independent of the initial abilities of students 
as possible. 

A process effect of. 18 percent compares quite favorably with the 
less then 5 percent that is currently claimed as the amount that can be 
attributed to school differences (Averch et al. , 1972; Coleman et al. , 
1966). Should this study find a process effect that is larger than 5 per- 
cent, it would be an important finding for restoring confidence in the no- 
tion that what happens in school does make a difference in whal^children 
learn. It is more likely that such an effect will be found by directly meas- 
uring the instructional processes that are operating at the classroom lev- 
el, rather than by measuring variables that characterize the school. 

Tables 6. 2 and 6. 3 are hypothetical outcomes at the next level of 
generality. They are relevant to the question of whether the guiding model, 
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• Table 6.1 , 
Partition of Variance for Gain in Grade 1 Math Achievement 
(Illustrating unique contribution of classroom processes) 

Source Proportion of Variance 

Initial Abilities 
Classroom Processes 
Commonality 
Error 

Total Criterion Variance 1.00 



. 28 
. 18 



22 
32 



This table, as well as" the other tables in this section, pre- 
sents the hypothetical results of an analysis of grade 1 math- 
ematics data. Similar analyses should be conducted and re- 
ported on grade 3 mathematics data, and grade 1 and grade 3 
reading da*ta. 
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Table 6. 2 



Partition of Variance for Gain in Grade 1 Math Achievement 
(Illustrating weak contribution of program variables) 



Source 
Initial Abilities 
Support Variables 
Program Variables 
Commonality 
Error 



Proportion of Variance 
.28 

.15 \ ■ 
. 03 
. 22 
. -32 



Total Criterio'n Variance 



1. 00 



Table 6. 3 



Partition of Variance for Gain in Grade 1 Math Achievement 
(Illustrating strong contribution of program variables) 



Source 
Initial Abilities 
Support Variables 
Program Variables 
Commonality- 
Error 



Proportion of Variance 
. 28 
. 08 
. 10 
. 22 
. 32 



Total Criterion Variance 



1. 00 
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which asserjbs that prograir^ variables are essential to the explanation of 
pupil learniAg, is supported by the results. The results in Table 6.2 ^ 
do not provide strong support for the model, since the contribution of 
program variables is small. The outcomes illustrated in Table 6.3, 
on the other hand, do . seem* to justify the model. 

However, several considerations must influence what has to be 
a subjective decision regarding the usefulness of the program variables. 
Assviming that smaller increments of explanation are useful when larger 
proportions of total outcome variance are accounted for, whereas only 
larger increments of explanation can impress^when the best model is 
weaker, then one must view the uniqueness for the program domain in 
the perspective of the total of all uniquenesses and corrumonalities, or 
the "total variance explained. " Small improvement of a strong prediction 
system can be worth considerable cost. Only substantial improvement 
of an inherently weak prediction system may be worth the same cost. In 
addition, it is necessary to know which of the program variables seem to 
be most involved in producing the uniqueness for the program domain. 
Some pro,^ram variables are more amenable to policy manipulation, or 
mc^re economical to manipulate, than others. If the indicated manipula- 
tions are easy and cheap, they may be justified by a relatively small unique 
ness. Also, it is important to know what values of initial abilities and sup- 
port\variables seem to combine with what specific values of prqgram vari- 
ables to produce the best results, as well as wiiich combinations are to 
be avoided. It must be remembered that the best model for the data is - 
likely to be an interactive one that does not i^^d out the. same rewards 
for a given value of a prograrA variable over all values in the ranges of 
support variables and initial abilities. This interaction can be repre- 
sented in the model by scales fabricated as cross-products of other scales. 
Another consideration concerns which aspects of a complex Criterion per- 
formance seem to be most and least influenced by the program variables. • 
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In mathematics, for example, if simple computing is influenced* but 
problem solving is not, policy makers may not value the learning that 
is influenced as much as they do the global concept of "mathematics." 
All this ^eans that there must be other tables that "unpack" the sum- 
mary of results in tables such as 6.2 or 6.3. 

Table 6.4 reports hypothetical results in terms of the four proc- 
ess constructs and illustrates how the commonalities themselves can be 
of interest. ' All but the motivators construct seem to be important 'in 
explaining gain in mathematics achievement. This lack of a unique con- 
tribution for motivators may be the result of unreliable measures or of 
confounding with other predictors, 'Whether or not confounding exists can 
be determined by an examination of the commonalities. As the table in- 
dicates, the motivators construct (3) is confounded with initial abilities 
(1) and opportunity (2), as well as being part of an in-common effect for 
all five of the predictor sets. Thus, although motivators may be impor- 
tant ip influencing achievement *in mathematics, the correlation between 
motivators and the other predictors makes it impossible in these par- 
ticular results^to identify a unique effect for motivators. 

Table 6.4 also illustrates the results of the analysis that would 
be performed if one defined individualization in terms of the structure _ 
and placement construct. The Table 6.3 analysis of the combined effect 
of the program variables represents the test of individualization that we 
propose, since we defme individualization in terms of the program vari- 
ables in both the structure and placement, and instructional events con- 
structs. The analysis reported in Table 6.4 distinguishes between these 
two constructs; this hypothetical illustration shows that both constructs 
are important predictors of student achievement. 

Table 6. 5 illustrates the most detailed leve.1 for unpaclling the 
summary statistics. Hypothetical uniquenesses are presented for the 
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Table 6. 4 



Partition of Variance for Gain in Grade J Math Achievement 
(Illustrating contribution of the four process constructs) 



Source 

1. Initial Abilities 

2. Opportunity 

3. Motivators 

4. Structuire and Placement 

5. - Instructional Events 
Commonalities 

1 and 2 

1 and 3 
. 1 and 5 

2 and 3 

4 and 5 ^ 
1, 2, 3, 4, and 5 



Error 



Proportion of Variance 
.28 
.08 
.00 
.06 
. 04 

. 04 

.03 - ^ 
. 05 
. 02 

.01 , 
. 07 

32 



Total Criterion Variance 



I. 00 
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Table 6. 5 

Variables with Full Model for Partition Reported in Tables. 6. 3 and 6.4 



Initial Ability Uniqueness 

1. Classroom means . 27 

Z. Classroom variance ^ .01 

Support Variables 

3. Amount of time available to learn 

subject matter .03 

4. Curricular overlap * .05 

5. Curricular motivators , OO 

6. Interpersonal motivators . 00 

Program Variables 

7. Specification of objectives .^3 

8. Matching of students and curriculum . 00 

9. Sequencing and pacing of instruction . 03 

10. Grouping . 00 

11. Management information . 00 

12. Cognitive teaching to individuals or 

small group .01 

13. Cognitive teaching to whole class , 00 

14. Indirect teacher behavior . 00 

15. Quality of teaching techniques .03 



One of the process variables, "quality of materials, " can only be meas- 
ured on progr§.ms that include diagnostic tests and procedures for indi- 
vidual decision making. A separate analysis should be run with class- 
rooms operating such programs. This analysis would make it possible 
to determine the variation in instructional effectiveness amoAg materials 
produced for individualized programs and the effect of this variation on 
achievement gain. 
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15 predictors of classroom gains in mathematics learning for 200 fir-st- 
grade classrooms. These uniquenesses indicate, that certain variables 
seem to be .irrelevant for this criterion, or at least make no independent 
contribution to its prediction. The uniqueness for initial abilities sug- 
gests that what a class holds for knowledge and ability at the beginning 
of the instructional year strongly prefigures what it is likely to gain dur- 
ing the year as new knowledge and-abilities. The uniqueness for the two 
opportunity measures ("amount of time available to learn subject matter'* 
and '»curricular overlap") reaffirms the notion that pupils learn what they 
are taught to the extent to whidh time is devoted to that teacher-learning 
effort. The uniquenesses that describe the most payoff for the individ- 
ualization of instruction are for "specification of objectives, " "sequencing 
and pacing of, instruction/" and "quality of teaching techniques. " 

The computer program tha.t we recommend be used for the com- 
monality analyses (Veldman, 1975) computes all possible commonalities, 
but only reports those larger than . 01, since the number of combinations 
becomes quite large as the number of variables increases. Commonali- 
ties smaller than .01 can also be quite important, however. It is pos- 
sible that some variables can only occur in combination with others, and" 
thus can only exhibit an in- common effect. These variables should not 
be dismissed as unimportant if a unique contribution is not found. For 
example, "matching of students and curriculum" (v^ariable 8) shows no 
unique effect, but might well reveal a joint effect with "specification of 
objectives" (7) and "sequencing and pacing of instruction" (9). This joint 
effect would be indicated if high scores on (8) only occurred in« combina- 
tion with high scores On (7) and/or (9). Attention to regression coeffi- 
cients, alone would not reveal this type of effect. 

In inferring the relative importance of the predictors from their 
uniquenesses, one must also consider the reliability with which the proc- 
ess variables were measured. It is possible that a variable may not appe 
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to be important simply because it was less reliably measured. Some 
investigators have suggested the possibility of building reliability infor- 
mation into the estimation of regression coefficients (e.g., Wiley, 1973). 
We have suggested that the data be analyzed using commonality analysis, 
but, clearly-> alternative analysis Strategies should be explored by the 
contractor. ' 

Variance is a bread-and-butter idea for educational researchers, 
and explanation of criterion variance is the essence of successful curricu- 
Ixxm research for them. Persons in Congress cannot be expected to share 
this viewpoint. The abstract notion of explained variance can be grounded 
in real terms for policy makers by developing examples of predicted out- 
comes for various vectors of predictor values by means of the raw- score 
regression equation. For a constant level of initial abilities and support 
variables, various levels of program variables can be demonstrated as* 
possible manipulation strategies. The disclaimer that policy manipula- 
tions are not likely to have guite as much effect as predicted from the 
regression equation has to be attached, but these demonstrations may 
provide the best advice to policy makers the study can afford- If inter- 

\ action variables have been validated, the demonstrations will have to b.e 
\ ^ . ■ 

made for several levels of the interacting variables. 

An auxiliary to the model for classroom learning should be a model 
for the variability among clas srooms^n the program domain. This model 
' (technically a partitioning of canonical redafidancy) would explain from 
initial abilities and support variable^^some of the program variability. 
It would help policy makers to understa>nd who gets what in school under 
present sociopolitical realities, and migPHUlu^inate policy options. 

Persons in Congress no doubt will be most interested in the ef- 
fects of program variables on the gain in achievement, as estimated fronl 
classroom means. Yet program variables may have socially significant 
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effects on the shape of classroom learning distributions, as well as on 
the means. Of Fisher's four descriptors (Lohnes, 1972) of classroom 
learning distributions, the variance and skewness have been found to 
be the most*useful in describing the shape of the distribution. Suitable 
graphic demonstration of the program effects might be useful. Com- 
puter plotting graphics will be needed ^to reveal the shape implications 
of various vectors of values for the raw regression 'equation. It would 
be remarkable indeed if the study were to document graphically that in- 
dividualization, without segregation of low ability youths, tends to shorten 
the left tail of learning distributions in the manner required for "mastery 
learning. " Elongation of right tails would indicate the nurturance of un- 
usual talent. It will also be possible to explore whether or not instruc- 
tion that adapts to individual differences in learners increases or de- 
creases learner variability. 

The Individualized Instruction Study is a study of what children 
learn in the course of a school year. It is vital that the criterion vari- 
ables represent what children learn in the year of the study, rather than 
what they had learned in six to eight years of life prior to the study year. 
It is well known that strong models for initial abilities can be made. The 
only way to provide a criterion that represents what children learned in 
one year, rather than what they know from all their years of learning, 
is to subtract what they could do at the beginning of the year from what 
they could do at the end of the year. This change in performance is the 
natural criterion language for the study, and persons in Congress will 
appreciate it, even if statisticians sometimes decry it. It should be 
noted that most of the criticism to which cliange scores have been sub- 
jected over the years has been appli,ed to situations where the individual 



Recently, however, this question has been reexamined, and we do not 
now feel quite, so apologetic regarding the use of gain scores. See, for 
example. Overall (1975), Kenny (1975), and Richards (1975). 
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pupil is the unit of analysis. The change scores proposed for this study 
will be changes in classroom means. Presumably, the reliability of 
these change scores will be reasonably high. Whatevet the technical 
characteristics, they represent what Congress wants to know about, 
Sonje risks have to be taken. 

Program, profiles . Another analysis that should be conducted as 
part of the main investigation of classroom processes involves a com- 
parison of variables that prove to be effective in supporting learning. 
For example, taking the uniquenesses reported in Table 6. 5 as an indica- 
tion of the most effective process variables, values on variables 3, 4, 
7, 9, and 15 could be averaged for all classrooms that are using a p?Lr- 
ticular program. These results, when properly interpreted, could have 
important implications for schools /that must decide which program to 
implement. 

Figure 6. 1 illustrates the profiles for three hypothetical prpgrams 
(A, B, and C). The location and shape of the distributions for classrooms 
using these three programs are represented by schematic box plains (Tukey 
1972). The horizontal line cutting through each box represents the median 
of the classrooms operating that program, the height of the box r|epresents 
the 25th and 75th centile points, and the vertical lines extending from the 
box indicate the range of observed values. The differences on '^time, ^' 
although important in explaining achievement^ are not relevant to the se- 
lection of a program, since the amount of time allocated to reading and 
math can be manipulated independent of the instructional program that 
a school uses. The "overlap" measure needs to be considered at least 
to the extent required to understand unusually low scores. A low score 
would suggest that a program does not include many of the objectives in- 
cluded in the achievement test. If the omitted objectives were viewed as 
important, then a program that is low on "overlap" would not be an appro- 
priate choice. Of the five effective process variables, "objectives" and 

: \ 
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"pacing'* are'the ones that are most directly a function 6f the program 
being used, and thus are important in program selection. In this illus- 
tration, program B has the highest scores on these two important fea- 
tures. The fifth variable, '.'teachmg, may be related to the kinds' of 

teacher behavior that the program or "its accompanying training package 

♦ * 

encourages (or allows), or it may be related only to the characteristics 
of individxial teachers. Thus, this variable should nat-be given too much 
emphasis in selecting among available programs.' A study that simply 
contrasted achievement gain for these three programs could not provide 
this kind of information*. Program A, for example, might app^ear to be 
as effective as the others ^Drimarily because the teachers who happened 
to be using it devoted time to mathematics to the exclusion of instruction 
in other important skills. Although we do not expect that any single pro- 
gram incorporates all of the features of effective instruction, this type 
of program profile analysis could indicate which programs appear to be 
the best bets. 

Instructional Setting Contrast 

In addition to showing the extent to which classroom processes 
can affect children's learning and the -relative effectiveness of different 
individualizing procedures, it is important for the Individualized Instruc- 
tion Study to attempt to show that 19% ability children can perform as well 
in classrooms in which effective processes are operating as they perform 
in Title I tutorial situations. For this aspect of the study, the general 
approach is to identify tutorial programs in District Survey I that are in 
the satae geographic region that is under investigation in the Individualized 
Instruction Study. Fr<t)m our examination of Title I files in Pennsylvania, 
we Estimate that most communities have Title I programs that would be 
suitable for including in this contrast. Only 100 tutorial students in grade 
1 need to be involved, since the student will be the unit of analysis. These 
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/students shoiild be included in both the. fall and spring testings. Tutorial 
iprograms will not^ of course, receive intensive process observation, but 
they should be Visited during the school yefar to make sure that they are 

' consistent with available descriptions. It will be important to determine 
if tutorial, work is instead of or in a'ddition to regular classroom instruc- 
tion in reading and/or mathematics . If possible, the tutorial participants 
should be selected from programs in which the tutorial work displaces 
classroom instruction in the same subject. 

After the process data on classrooms in the Individualized Instruc- 
tion Study are analyzed, the tutorial children should be matched on abili- 
ties with 100 first-grade students. from classrooms. that are revealed to 
be high on effective process measures. The contrast between the two 
groups of students could be a simple covariance design. The expecta- 
^ tion is that low ability children will perform at least as well in a regular ^ 
classroom as in a Title I tutorial program, particularly if that classroom 
was rated high on the process measures shown to be effective in the main 
analysis. It will also be possible, without cpllecting additional data, to 
contrast students , in the tutorial mode with students from classrooms 
that exhibited varying degrees of effective proce ss . This ^contrast would 
allow one to examine just how effective classroom process must be in 
order to be at least as effective as, if not better than, the very expensive 
tutorial mode. ' • ' 

Subsequent Analyses 

Given the rich dataset that will be collected in the study that we 
are proposing, numerous analytical approaches are possible and should 
be tried after the main analysis is completed. What any good design must 
do is ensure that analytical schemes are available for answering the ma- 
jor questions of the study, but much more investigation will be possible 
and desirable after the main effects hav6 been e stablished. For example, 
% "* ■ 
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it might be desirable to rerun the main analyais, including in the esti- 
mate of achievement gain only those students who would be considered 
eligible for compensatory programs (using a uniform criterion of elig- 
ibility across all classrooms) and adding the mean initial' ability level 
of the eiitire classroom to the predictors. If the classroom ability level 
proved to have a positive impact on the achievement of eligibles, it would 
suggest the importance of heterogeneous grouping in facilitating the de- 
velopment of low achieving children. Interaction terms could also be 
tried. To do so requires introducing the product of two variables as an 
additional predictor. It is possible, for example, that heterogeneity is 
effective only in the presence of good structure and placement. 

One might also conduct an analysis in which the program that 
each classroom is using is included as a dummy variate. If the result- 
ing set of variates shows an important program effect in the presence 
of our process measures, then the conclusion would be that there are 
important differences among programs that are not being captured by 
the process measures. That finding would have important implications 
for program selection. as well as indicate the need for revision of the 
classroom processes model. 

I 
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Section 7 

Project Organization^ Staffing^ and Schedule , 



Organization and Staffing 

Any organizational structure that we might suggest for the Indi- 
vidualized Instruction Study will be and should be modified by 'the con- 
tractor to reflect the nature of the contracting organization and the com- 
petencies of the individuals who will Be responsible for major functions. 
If any subcontractors participate in the study, other modifications may 
have to be made. For these reasons, only a very general project struc- 
ture i? presented (Figure 7, 1), The necessary qualifications of each of 
the key individuals included in this structure are briefly described below, 
along with the functions that they will perform. 

The Project Director must have extensive experience in the ad- 
ministration of large-scale research projects. In addition, this indi- 
vidual must have special competencies in the areas of field research in 
education, statistical analysis, evaluation methodology, aiid instructional 
design. The Project Director will haVe overall responsibility for the study, 
with direct responsibility for report preparation and coordination with NIE, 
Further, he or she may also be qualified to perform one of the other func- 
tions specified below. 

There are at least three reasons why it is very important that an 
exceptionally well qualified individual direct the project. First, the In- 
dividualized Instruction Study involves the collection and analysis of enor- 
mo\is amounts of data, the management of . a large number of project per- 
sonnel, and coordination with many outside individuals and groups, Sec- 
ond, the stucfy must be completed in only 18 months, which is a short 
period of time for such a mammoth undertaking. Third, and perhaps 
mpst important, the results of the study will likely impact on thousands 
/of children across the nation. Thus, the study must be \^ell executed 
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and well reported. To achieve this requires direction by an extremely 
knowledgeable and experienced individual. 

The Director of Data Collection must have previous experience 
in field research in education that included the administration of stan- 
dardized tests and the conduct of classroom observations. He or she 
must be familiar with the types of problems that can occur in a large- 
scale data collection effort and with techniques for solving these prob- 
lems. This individual's responsibilities include supervision of the. 
School Coordinator, Testing Coordinator, Field Coordinator, and Site 
Coordinators, as well as the reporting and preliminary interpretation of 
classroom data. 

The School Coordinator will serve as liaison between study per- 
sonnel and participating school districts. He or she will be responsible 
for identifying and securing the cooperation of specific schools and for 
making all necessary arrangements with state and local school adminis- 
trators for testing, videotaping, and interviewing to take place in the 
classrooms (e.g., obtaining school board permission).. Because the 
School Coordinator must work closely with educational administrators, 
it is strongly recommended that this individual have experience working 
in a public school system, preferably as a district superintendent or prin- 
cipal of a large school. 

The Testing Coordinator will be responsible for all activities re- 
lated to the administration of achievement tests in the fall and spring, 
including ordering and disseminating tests, coordinating the hiring and 
training of test administrators, arranging for test scoring, and ensuring 
that adequate quality control procedures are followed in test administra- 
tion and scoring. As indicated previously, the test administrators, pos- 
sibly substi/tute teachers, should be hired specifically for this study. 
Approximately iOO administrators will be required at $500 per person. 
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The Field Coordinator will supervise all activities related to 
the collection of process data through interviews, curriculum analysis, 
and videotaping. Examples of these activities including disseminating 
all necessary data collection materials,^ arranging for the purchas-e and 
maintenance of videotaping equipment, obtaining curriculum materials 
for analysis, and coordinating the hiring and training of the field staff. 
This field staff will conduct in-class teacher interviews, analyze^ur- 
ricula in use in participating classrooms, videotape in-class activities, 
and code all process data. The number of individuals and person-months 
of effort will depend on the functions that the contractor assigns to each 
person (e.g., interview teachers, or interview t^jpachers and also code 
the data), the time schedule adopted, the quality lontrol procedures im- 
plemented, etc. Our estimate is that a field stafflwill be required of at 
least 5 full-time individuals for the duration of theVstudy and 35 full- and 
part-time individuals for varying amounts of time up to 15 months. 

The Site Coordinators will serve as liaisons between the project 
staff and school principals and teachers at each site. They will, for 
example, arrange the scheduling of in-class interviews and videotaping, 
monitor and assist in the collection of both process and achievement data, 
and respond requests for information about the study from school per- 
sonnel or parents. Approximately 40 Site Coordinators, possibly school- 
related personnel, will be required. It is anticipated that over the course 
of the study they will work the equivalent of 8 person-months each. 

The Director of Data Processing and Analysis must have exten- 
sive experience in supervising the processing and analysis of large amounts 
of education-related data, with special competencies in statistical analy- 
sis and evaluation methodology. His or her staff will include a full-time 
programmer, a small group of coding clerks who will prepare data for 
keypunching, -and a keypuncher. In most organizations, coding and 
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keypunching are done by a support group. Thus, the coders need not be 
full-time members of the .project staff. 

Schedule 

A proposed schedule of project activities and products to be de- 
livered to NIE is presented in Table 7. 1. In preparing this schedule, an 
attempt was made to specify products that would be the direct result of 
ongoing study activities (e. g. , a confirmed list of participating schools, 
principals, and teachers), rather than products that would have to be 
prepared solely for the purpose of keeping NIE informed. Only two ma- 
jor reports are required--a status report in February 1977 and a final 
report in July 1977. 
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Date 

February 1976 
March 1976 



April 1976 
May 1976 



June 1976 



July 1976 



• Table 7. 1 
Project Schedule 



Activity 

Complete sampling plan. 

Prepare tentative list of 
s chobls, principals, and 
teachers. 

Secure cooperation of par- 
ticipants. 

Prepare tentative list of 
curriqula in each class- 
room.. Order these ma- 
terials. 

Begin making arrangements 
for fall data collection (e. g. , 
staff, instruments, eq\iip- 
ment, schedule, parental 
permission). 

Prepare practice videotapes. 

If necessary, request per- 
mission from NIE to admin- 
ister standardized test other 
than test previously agreed 
upon. 

Conduct workshop for data 
collection personnel. 



Place orders for tests. 

Select tutorial sample for 
instructional setting contrast. 



Product to Be 
Delivered to NIE 



Tentative list of 
participants 

Confirmed list 
of participants 

Tentative list of 
curricula to 'be 
analyzed 



Data collection 
schedule 



Agenda and sum- 
mary of workshop 
presentations 



ERIC 



89 



86 



Table 7. 1 (continued) 



Product tQ Be 

Date Activity Delivered to NIE 

August 1976 Conduct workshop for local Agenda and sum- 

site personnel at contractor's mary of workshop 
office, presentations, or 

workshop attend- 
ance by NIE repre- 
sentatives 

Complete arrangements for 

fall data collection. ^ 

Train test administrators 
and curriculum experts. 

September 1976 * Obtain class rosters. Class rosters 



^ Prepare final list of curricula 
in each classroom. Order any 
materials not previously or- 
dered. 

Begin curriculum analysis and 
complete overlap estimates. 

Administer achievement tests. 

Schedule make-up tests. 

Cross-check tests against class Estimate of per- 
rosters. centage of missing 

^ data by classroom 

Begin test scoring. 



Final list of cur- 
ricula to be ana- 
lyzed 
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Table 7. 1 (continued) 



Date 



October 1976 



November 1976 



December 1976 



Activity 

Complete administration of 
make-up tests. 

Complete test scoring. 

Complete training of video- 
tapers and interviewers. 

Begin videotaping and^nter 
viewing. 



iculi^^ analysis. 



Continue curr 



Complete videota]>ing and inter- 
viewing (do not tape or inter- 
view two days before or after 
Thanksgiving receps). 

Analyze fall test results. 



Product to Be 
Delivered to NIE 



Results of fall 
testing for both 
main study stu- 
dents and tutorial 
sample 



Train interview coders ^.nd 
begin coding of interview data. 

Continue curriculum analysis. 

Complete curriculum analysis. 

Train videotape observers. 
Begin coding of videotape data. 
Continue coding of interview data. 



Results of cur- 
riculum analysis |^ 
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Table 7. 1 (continued) 



Date 

Jantiary 1977 
February 1977 



March 1977 



April 1977 

May 1-17, 1977 
May 18-31, 1977 
June 1-15, 1977 



June 16- 
July 31, 1977 



Activity 

Complete coding of video- 
tape and interview data. 

I Prepare status report and 
outline of final report. , 



Complete schedule for $ec- 
ond round of videotaping 
and interviewing. 

Complete videotaping and 
interviewing (da not tape 
or interview two days be- 
fore or after spring break).. 

Complete coding of videotape 
and interview data. 

Complete scheduling for spring 
testing. 

Construct final dataset minus 
outcomes. 

Administer achievement tests. 

Complete test scoring. 

Complete data analysis. 

Select individualized sample for 
instructional setting contrast. 



Conduct instructional setting 
contrast. 



Product .to Be 
Delivered to NlE 



Status report and 
outline of final re- 
port 



Prepare final report. 



Final report 
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In-Class Teacher Interview Questionnaire 
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Name of Intervie we r 
Date 

School District ' * 

School 

Grade 

Teacher ^ s Name 

Years of Teaching Experience 

1. How many students are enrolled 
in this cjassroom?^ 

2. How many students are present, 
today? 

3. What is the average yearly 
attendance? 

# student days 
# students 

4. How many school days are there 
in the year? 

5. How many children have trans- 
ferred into this room? 

6. How many children have trans- 
ferred out of this room? 

7. * How many* adults are usually in 

this room? 

8. How many minutes are in the 
school day? (8e - 8b - [8d - 8c]) 

8a: What time do students arrive? 

8b. What time do classes begin? 

8c. What time are students dis- 
missed for lunch? 

8d. What time do students return 
from lunch? 

8e. What time are students (^^^^ 
missed for the day? 

/ 
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• 9. How many minutes a day are spent 
in mathematics instruction? 

10. How many minutes a day are spent 
in reading or reading -i?%iated 
instruction? 



11. How much homework is assigned in mathematics ? 

^ None A Lot 

12 '3 4,5 

12. How much homework is assigned in reading? 

None • A- Lot 

1 2 p 3 4 5 

13. How frequently is peer tutoring used in this classroom? 

Very - 

Never Frequently 
1 2 3 4 * 5 

14. Are games and/or contests used in the teaching of math or reading 
in this classroom? 

Very 

Never ^ I^requently 
1 2 3 '4 5 

15. Do students score or grade their own te^ts ? 

■ I V Very 

Never ^ Frequently 

1 2 , 3 4 5 

16. Do students decide how they want to work- -alone or in small 
groups or teams ? 

Very 

Never ' ' Frequently 

1. 2 3 4 * 5 

17. - Do students decide whom to sit next to ia math or reading? 

Very 

Never ' . Frequently 

.1 2 3 4 5 

18. Rate the reading materials generally available.in the classroom 
as to how interesting the students find |:hem. 

Not . . \ V-ery 

Interesting \ Interesting 

- 1 2 3 4 5 . 
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19. Rate the mathematics materials generally available in the class- 
" room as to how interesting the students find them. 

Not v^^y 

Interesting ^ ■ Interestiiig 

1 2.3' 4 5 

20. Within a single mode of reading instruction such as pencil and 
paper, rate the variety of instruction format. 

* \ Large Amount 

. ' Variation °^ Variation 

1 2 3 4 5 

21. Within a single mode of mathematics instruction^ such as pencil 
and paper, rate the variety of instruction format. 

. • Large Amount 

Variation ' . . of Variation 

. 12 3 ^ 4 ^ 5 

22. Check the various modes of instruction in use in reading. 



Audio tapes 
Workbooks or sheets 
Other books 
Film strips 
Additional (specify) 



23. Check the various modes of instruction in use in mathematics. 



Workboo^cs or sheets 
Flash cards 
Game s« 
Manipulative s 
Audio tapes 
Film strips 
Additional (specify) 



24. On the average, how long after a student has done a specific' 
academic task does he/she receiye information about the cor- 
rectness of his/her performance? 

How Many 

Minutes 

Hours ^ 1. - 



Days 
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•25. Wh^^types of objectives are present in the reading curriculuil? 
(checK^one) , * ' • . 

General 

Specific 

Both general and specific 
No. objectives at all 

26. If any specific objectives are present, rate the reading curriculum 
on the clarity of its specific objectives. 

Unclear , * d^^^ 

1 2 3 4 5 

27. When a new objective is presented in the reading materials, which 
of the following appear? (check appropriate lines) ' 

A written statement of the objective 
, An example of the new objective 

A range of examples of *the new objective 

Practice problems for th^ student to complete that are 

different from the example * 
A non-example 

28. • Rate the reading curriculum as to how closely the available 

materials match the stated objectives. 

No Close 
Match ^ • ' . Match 

1 2 3 , 4 5 . 

29. At the beginning of the year, how do you decide where a child 
should ^tart in reading or in which reading group he/she belongs? 
(check appropriate lines) * 

By starting at the beginning of the text or series 
By the results of a standardized test 
By the results of a curriculum-designed test' 
By the results of a test that you developed 



From the placement of the child at the e;n.d of the previous year 

From the .wishes of th^ child 

Other, please specify. 



30. ^ On the averalge, how frequently do you think a child starts on a new 
reading objective (topic) ? (check one) 

Once 'a day 



Twice a Week 
Once a week 



Once every othej* week. 
Once a month 
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'31/ On the** average, what amoixnt of reading time da you think* a child 
spends reviewing material that he/she has already learned? ^ 



32. What types of objectives are present in the mathematics curriculiim? 

• General * " . ^ 

Specific 

Both general and specific ^ 
\ No 6bjectives at all - , 



33. If any specific objectives are present', rate the mathematics 
curriculum on the clarity of its specific objectives, 

• Unclear ' ^ Clear 

1 2^3 ,4 5 

'34. When a new objective is presented in the mathematics materials, 
which of the foUbwing appear? (check appropriate lin.es) 

* • 

A written statement. of the objective 

,An example of the new objective 

A range of examples of the new objective 4 ' 

Practice problenis for the student to complete th^are 

different fi;pm the example 

A non- Example - 

35. Rate the mathematics curriculum as to how closely the available 
. materials match the stated objectives. 

No . , Close 

"Match ' Match 

1 ^ ' 2 3.4. 6 

36. At the beginning of the year, how do you.decide where a child 
should start in njiathematics ? (check appropriate lines) 

By starting at the" beginning of the texb or series 

By the results o£ a standardized test 

By the results of a curriculum-designed test . 

By^'the reWlts of a test that you developed 



From tl^ placement of the. child at the end of the previous year 
From the wishes of the child , 

Other, please specify ^ \ _l. 



37. On the average, how frequentlyfido you think a child starts on a new 
mathematics objective (topic)? (check one) 

Once a day 



Twice a week 
Once a week ' 



Once every other week 
Once a month 
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38, • Orf the average, what amount of mathematics time do you think 

a( child spends reviewing material that he/she has already - 
learned? _^ 

39, Is there a gystematig way of assessing student initial abilities built 
into the reading curriculum (e. g. / are some types of placement 
tests provided or is a procedure recommended) ? ^ 

40, Do you use it? 



41. ' Is there an informal way that you use to assess student initial 

. abilities in reading ? ^ . 

42. ' If so, please specify. . 



43. Is there a systematic way of assessing student mastery of 
specific skills built into the reading curriculum (fe. g. , are tests 
provided)? ^ 

44. Do you use it? _^ 



45. ftate the reading curriculum according to how clear it is what . 
the next unit should be at the completion of a unit of material. 

"""^ Very 

Unclear " ' , Clear 

1 ' 2 3 ^4 5 

46. How"frequently do you skip arouad in the reading sequence or text? 

Never / , O^^^ 

1 2 ; ^ . 3 4 5 

47. How frequently do you create your own reading materials in-order 

• to improve the sequencing that is specified in the reading curriculum? 

Never , Oit^n 

1 2 ^3 4 5 

m 

48. Is there a systematic way of assessing ^tudent initial abilities 
built into the mathematics curriculum (e. g. , are feome types of 
placement tests provided or is a procedure recommended)? 



49. Do you use it? 

50. Is there an informal way that you use to assess- student initial 
abilities in ma'th.ematics ? 

51. * If so, please specify. ^ , 
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52. Is. there a systematic way of assessing- student mastery of 

specific skills built, into the mathematics curriculum (e. g. , are 
tests provided)? • 



9 



53. Do you use it 

54. Rate the mathematics curriculum according to how^ clear it is 
what the ne-xt unit should be at the completion of a unit of material. 

- ' Very 

TT . 1 . ' Clear 

Unclear ■ ' ^ 

1 - 2 3 ^ 4 5 

55> How frequently do you. skip around in the mathematics sequence ^ . 

' or text ? • ■ 

56 • How frequently do you create your own mathematics materials in 

order to improve the sequencing that is specified in the mathematics 
curriculum? — 

Never ' , Often 

.1 2 ^ ^3 ^ ^ 4 5 ^ 

57. Who decides what skill or concept the student will work on in 
reading? -(check most appropriate lines) ' ^ 

Parents ^ 

Teacher 



Curriculum 
Child ' 



ti/e 



58. Who decides what skill or concept the student will work on in 
mathematics ? (check most appropriate lines) 



Parents 

Teacher 

Curriculum* 

Child • • . / 

59. If a/student does not pass a test in reading, what dp you usually 
do? {check appropriate lines) *^ \ 

Tutor 

Give special work^ 

Continue on ^ 



Give extra homework 

Repeat the material cove-lfed 
Other, please specify 
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' ^60. If a student does not pass a test in mathematics, what do you 
usually do? (check appropriate lines) 

- Tutor. ^ -1,^ f 

Give special work . . 

Continue on , 

* Give extra homework 

Repeat the material covered ; ^ 

Other, please specify 



6 la. If a student has .been worlcing^for several days on a difficult con- 
cept-or skill in reading and seems to be making no progress 
toward mastery, what do you do? 



6 lb. If a' student has been working for several days on a difficult con- 
cept or skill in mathematics and seems to be making no progress 
toward mastery, what do you do? 



/ 



62. J^re the children self-paced in reading? 
Yes 

No ' . 



63. Are the children self-paced in mathematics ? 

'Yes 

No 



64. Over the last two months, what is the largest and. smallest num- 
ber of units completed in reading? 

^ Largest Smallest 



65. Over the' last two moMhs, what ife the largest and smallest num- 
ber of units completed in mathematics? 

Largest Smallest 



66. In reading, in what size group are children receiving instruction' 
Whole class . 

Individual ' ■ _ 



Small §roup 

How many groups are there? 

How many children are in each group? 
How often are groups re-formed? 
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• 67. In mathematics, in what size group are children receiving instruc- 
tion? 



\Vhble class 

* Individual 

. Small group 

How many groups are there? 

How many children are in each group? 

How often are groups re-formed? ' 
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Overlap 

Data. The following information will be used in calculating the curriculum 
• overlap. It should be collected in the fall. 

What is the name(s) of the math. 

series being used? ^ 

Are other texts being used ? . 

If more than one series is being 

used, report names of all pub- ^ ^ 

lished series. . 



Do special groups use them 
(e. gt , advanced or remedial 
groi^ps)? 

If ^o, who uses them? 

Do^ you use. the extra series to 
•-^'j^atch'* gaps in the existing 
oiie? ' 

•X^ist the topics that are taught 
using supplementary ijiate rials 
(e. g. , division of fractions, 
the number line). 

What is the name(s) of the reading 
series being used? 

Are other texts being used? 

If raore than one series is being 
used, report names of all pub- 
lished series.^ 

Do special groups use them 
(e. g. , advanced or remedial 
groups) ? 

If so, who uses them? 

Do you use the extra series 
to '^patch" gaps in the existing 
one ? 

List the topics that are taught 
using supplementary materials. 
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The following info rmatiou will also be used in calculating the 
^curriculum overlap. It should be collected in the spring s 

70, Teacher's estimated percentage of overlap in reading • 



71. Teacher's estimated percentage of overlap in mathematics 
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. In-Class Teacher Interview Questionnaire; 

Directions for Obtaining Measures i 

* 1. Obtain this information from the number of studentfe' on the 
• . teacher's permanent and updated attend^ince -roster , t 

NOTE; Count the names; do not use self-reports, 

> 2. Count the number of students present in the room, and count 

* " the number oh th6 attendance sheet. If the two numbers differ, 

try to get clarification on the diffe'rence (e, g, , Johnny went to 
tutoring) and record the maximum. \ - ^ 

3, From the end-of-year attendance summaries, obtain the total 
number of school days attended by each^child and add them. 
Divide this sum by the total number of students. 

Include ; ,AlI children whp were present in the 

fall and spring. * ^ . 

Exclude ; Any child who was absent 'in either 
^ ^ the fall or spring and was also absent 

for more than 40% of the school year. 
Example : Child A. starts school on November 10 
arid is present every day thereafter - - 
include. Child B starts school on . ^ 

September 1, is enrolled through the 
.end of the school year, and attends less ^ 
than 50 days --include. Child G starts 
' , school on September 1 and transfers 
25 days later --exclude. 

NOTE; Do not ask this question ih the fall. 

4. Obtain this .information "from a school administrator or teacher. 
(Include only diys that children are in school- -exclude in-ser-yice 
days, -etc. ) *) . 

5-6. Obtain fl^is information from end-of-year records of the teacher 

or principal. . . , » . 

NOTE; Do not ask this question in the fall. 

7. Ask the teacher, count the adults, and discuss any differences. ^ 
Include student teachers if they are present more than 50% of 
the yea^r {e^ g. , all year, '3 days a week). ^ , . ^ 

8. Ask the teacher Items 8a-8e. Observe 8a-8c and resolve any 
differences. Calculate 8 from the set. of information obtain«^d. 
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9-10. Ask these questions directly to the teacher. 

11-12, Ask these questions directly to the teacher. 

None (1) means that no homework is assigned. A lot (5) means 
that houaework is assigned every night and at least equals the 
amount of time spent in the subjec{ during school hours. 

* 13. Ask this question directly to the teacher. 

Definition; " Peer tutoring is occurring if one student in the class 
teaches another student(s) in the same class a specific activity 
. or helps him/her to rehearse a specific skill* Look around the 
room and see if there are any gToups of two or more students 
working together. Ask the teacher if peer tutoring ever occurs. 
If you saw no evidence of it and the teacher says "no," record 1. 
If you saw no evidence and the teacher says "yes, " aS'k'when it 
occurs (e.g., subject) and ask for the most recent (or immediate 
future) specific case of one student tutoring another. If the 
answers are vague and completely nonspecific, record 1, but 
if the answers are specific and reasonable, ask-the teacher to 
rate how frequently peer tutoring occurs. If you saw evidence 
of possible peer tutoring and the teacher says "yes, " recall the 
incidents ^you saw. Ask if they were, in fact, cases of peer tutor- 
ing. If so, ask for a rating; if not, probe as above. If you saw 
' evidence of possible peer tutoring and t]ie teacher says "no, " con-^ 
firm this response by describing what you saw and asking what it 
was. 

14. Ask this question directly to the teacher. 

In this question and' the next, three questions ,/ 1 means never and 
5. means every day for both reading and math* Look around the 
room. If you see any use of games or contests, ask for a rating. 
If you don't, ask the teacher . If he/she says "no," recotd 1. 
If he/she says "yes, " ask for specllfic examples and when they 
werfe used, then ask for a rating. If he/she is vague or unclear, 
mark 1. ' ^ 

15-,17. Ask these questions di];ectly to the teacher. Use the same techniqul^ 
as -in Item 14. • 
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18-19. After reading these questions once, explain the following: 

In asking you to respond to these questions, we are trying to elicit 
your judgi^aent about how motivating the curricula are to the stu- • 
dents you teach. While we cg.n judge this from a theoretical point 
of view, you have invaluable practical information about how much 
, * ■ the children are int^erested or intrigued by the material contained 
in the text. ' Please rate the curriculum using your owti experience 
and considering such things as pacing, format> artistry, etc. If, 
in addition, you would like to comment on any part of the curriculum, 
please feel free to do so. Your comments will be read and vajAied, 

20-21. After reading these' questions once, explain the following; 

For these questions, we are asking you to make a similar judgment. • ^ 
However, for the ratings consider the main mode(s) of presenta- 
tion such as textbook or workbook, and consider the variation that 
occurs within that mode. Consider variation in patterns of pre- 
sentation, layout, and type of response required by the student. 
For example, a program that presents text followed by questions \ . 
for the majority of the instruction has little variation and would be 
considered a 1. On the other hand, a program that has text followed 
by a variety of activities, such as writing short answers, writing 
plays, and writing questions for other students to answer, has 
large variation in mode and would be cqnside^red a 5. 

22-23. Check all modes of instruction that are evident in the classroom 
and then ask the teacher for specifics about the ,jase of additional 
modes. 

24. Ask this question directly to'the teacher. 

• Observe actual correction practices and resolve any differences. 
See, for example, if students' work is corrected while they are 
doing it or immedi'^tely afterward. K so, give your answer in 
minutes and disregard the other time units. If class work is re- 
turned during the period, check to see when-it was turned in to the > . 
teacher. If it was turned in the previous day, record the number 
of days '^old'^ the work is. 

25. After reading this question once, explain the following: 
A general objective is a global, program goal. 

Examples of general reading objectives: . ^ 

1. The child will become an independent reader, 
choosing to read for pleasure. 

2. The child will be able to extract meaning from 
written text. ' 
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Use only one example on the first reading. The second e:3j:ample 
should be read only if the information is not clear to the teacher, 

A specific objective is one that is linked with the instructional 
materials and is most often in behavioral terms. 

Examples of specific reading objectives: 

1. The child will be able to decode all one-syllable 
words in his/her. reading text. 

2. The cl^ild will be able to read a short paragraph ^ ^ 
story (three to four aentences) and paraphrase 
the story. 

Again, use o.nly one example on the first reading. The second 
example should be read only if the information is^Jiot- clear to the 
teacher. 

TfEen ask the teacher if the .reading curriculum has general . 
objectives only, specific objectives only, * both general and 
specific objectives, ojr no objectives at all. 

26. If no specific objectives are present in the reading curriculum, ^ 
skip torltem 27. If there are some^ specific objectives in the read- 
ing curriculum, ask the following question: \ 

Consider the clarity of the specific objectives in the reading cur- . 
riculum. On a scale from 1 to 5, rate the clarity of the specific 
objectives. 

A clear objective is one that is focused and narrow. It precisely 
states what the child will know how to do oh completion of the 
materials. An example of a clear specific objective that ^yould 
be considered a 5 is: upon completing this \mit, the student can. 
pronotmce all one -syllable words of the CVC form. 

An unclear objective is one that is dispersed and broad. It does 
not describe precisely what the child will be able to do on completion 
of the materials. An example of an imclear specific objective that 
would be considered a 1 is: upon completing this tmit, the child - 
knows how to read one-syllable words. 

27. After reading this questiomonce, explain the following: 

A written statement of a reading objective may appear either in 
the teacher's manual or the child's text or in both. An "Example" 
of a written statement of a reading objective is: the child will be 
able to pronounce any one- syllable word with two vowels, the 
final vo.wel being a silent "e. " ' 
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An example of the new o bjective is; to present the child with a 
list of words that conform to this rule (sale, bale, male) and to 
pronounce the words, forvhim /her , ^ 

A range of examples covers the broadest extremes of the domain. 
An example is: sale, Joe, ate. 

Practice problems for the objective could be a list of words con- 
forming to the rule that the child must pronounce by himself /her- 
self, 

' A non- example for the objective is one outside of the domain. 

An example is: rut, not, near,' ^ 

28. Ask this' question .directly to the teacher. 

The materials match closely (5) if the contents of both the materials 
. and the objectives are identical. The materjLals do not match the 
« objectives (1) if the contents are not identical. 

29. Ask this question directly to the teacher.^ If he/she gives multiple 
an&wera or the answer is vague, approach it in reverse: 

Ask ""Who is in the top reading group?" Pick one of the list of 
names. Ask "How did Ann get placed in that group?" Repeat the 
question for two or three other children until you have the most 
commonly used criteria for grouping. 

NOTE: Do not ask this question in, the spring. 

30. Ask this question directly to the teacher. 

If the teacher is unclear as to the meaning trf an objective, define 
it: An objective is a single block of material or a topic that a child 
can learn in a fairly short period of time. It is often, but not always, 
followed by a test. 

If the teacher is still unclear, change the question so that a specific^, 
child is mentioned: "What is Althea working on? When did she 
start that? When will she finish?" 

Ask this question directly to the teacher. If the teacher is uncer- 
tain as to what isMne ant, explain the following: 

A student may, for example, work. on objectives four out of five ^ 
days and on the fifth day review skills that he/she has already 
learned. If this were the case, then one-fifth of the time would 
be spent on review and the amount of time recorded would be the 
length of one subject period. Another student may spend the first 
10 minutes of a 3 O-rninufe period reviewing previously leaded 
objectives. In this caae, the amount of time recorded would be * 
10 minutes. 
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Then ask tjie teacher to try again to estimate-the amovint of 
scheduled time that studejits, on the average, spend doing some 
form of review such as practice or drill. 

32. After reading this question once, explain the following: 
A general objective is a global, program goal. 
Examples of general mathematics objectives: 

1. The child will demonstrate problem- solving skills. 

2. The child will enjoy playing matheimtical games. 

Use only one example on the first reading. .^The second example 
should be read only if the information is noF clear to the teacher. 

A specific objective is one that is linked with the instructional 
^ materials and is most often in behavioral terms. 

E^camples of . specific mathematics objectives: 

0 1. The child will be able to add two one -digit .numbers. 
2. The child will be able to count to 100 by fives. 

Again, use only one 'example on the first reading. The second 
example should be read only if the information is not clear t6 the 
teacher. 

Then ask the teacher if the mathematics curriculum has general 
objectives only, specific objectives only, both general and specific 
objectives, or no objectives at all. 

33. If no specific objectives are present in the mathematics curriculum, 
skip to Item 34. If there are some specific objectives in the mathe- 
matics curriculum, ask th^ following question: 

As you dicj for reading, consider for mathematics the clarity of 
the specific objectives. On a scale from 1 to 5, rate the clarity 
of the specific objectives. 

A clear specific objective that would be considered a 5 is: given a 
series of one-digit, two -numeral addition problems in the vertical 
form, the child will be able to correctly calculate their sums. 

An unclear specific objective (1) might be: given a s^t of one-digit . 

ta-ddition problems, the child will understand how to find their sums. 

f V ... 
*- - 

# •. 

v.. 
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34. After reading this question once, explain the following: 

A written stateinent of a mathematics objective may appear either 
in the teacher's rrianual or the child^s text or in both. An example 
'af a written statement of a mathematics objective is: the student 
will be able to sum any two one-digit whole numbers. 

An example of the' new objective is: to present the child with a 
list of solved problems and conceptual methods for calculating 
correct answers to the problems (e.g., number line representa- 
tions of the solutions to 3+1=4, 4+1=5, 1+2=3). 

A range of examples covers the broadest extremes of the domain. 
An example is: 5+4=9, 8+9=17, 6+7=13. 

Practice problems fof the objective could simply be a list ^f un- 
solved problems for the student to complete. 

A non-example for the objective is one outside of the domain. 
An example is: 1-1 = ? . 

35. Ask this question directly to the teacher. 

The materials match closely (5) if the contents of both the materials 
and the objectives are identical. The materials do not match the 
'•objectives (1) if the contents are not identical. ' . 

36. ^ Ask this Question directly to the teacher. If he/she gives multiple 

answers or the answer is vague, approach it in reverse: 

Ask '^Who'isrin the top mathematics group?" Pick one of the list of . 
names. Ask ^^How did Ann get placed in that group?" Repeat the 
question for two or three other children until you have the most 
commonly used criteria for grouping. ^ ^ 

NOTE: Do riot* ask this question in the spring. 

37. Ask this question directly to the teacher. 

If the teacher is unclear as to the meaning of aii objective, define . 
it: An objective is a single block of material or a topic that a 
child can learn in a fairly short period of time. It is often, but 
not always, followed by a test. 

If the teacher is still unclear, change the question so that a specific 
child is mentioned: "What is Althea working on? When did she 
' start that? When will she finish ? " 
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38. Ask this question, directly to the teache,r. If the teacher is^imcer- 

* » • 

tain as to what is meant, eicplain the following: 

A student, for example, may work on new objectives four out, of 
five days' and on the fifth day review skills that he/she has already 
^earned. If this were the case, then one-fifth of the time-would be 
spent on review and ^he amount of time recorded would be the length 
of one subject period. Anotiter student may spend the first 10 min- 
utes of a 30-minute period reviewing previously learned objectives. 
In. this case, the amount of time recorded would l^^ ill minutes. 

Then ask the teacher to try a^aiii to,estimate the amount of 

scheduled time that students, on the average, spend doing some 

form of review such as practice or drill. ' ^ 

« • 

39-40. Ask Item 39 directly to the teacher. 

Record "y^s" ar "no. " If "yes, " tlien ask if this is the vjray most 
children are placed in the curriculum. If so, record "yies'^for 
Item 40 and retain "yes" for Item 39. If the answer for Item 46 
is "no, " record "no" for Item 40 and retain "no" for Item 39 or 
change the "yes" in Item 39 to "no" by lightly crossing out the 
"yes" and writing "no." 

41-42. Ask Item 41 directly to the teacher. 

If the teacher is uncertain, say: "Sometimes teachers don't always 
rely on the regular placement tests or they don^t have any placement 
tests to begin with. Do you use some other way of deciding where 
to start the child?" Then ask for the "specific method" in order to 
verify the answer to Item 41. If there are no specific methods, 
record the answer to 41 as "no. " If there are specific methods, 
record the answer as "yes. " • < ^ 

43-44. Ask Item 43 directly to the teacher. 

Record "yes" or "no. " If "yes, " then ask if this is the way in 
which it is decided that most children have mastered a particular 
skill, if so, record "yes" for Item 44 and retain "yes" for Item 
43. If the answer for Item 44 is "no," record "no" for Item 44 
and retain "no" for Item 43 or change the "yes" in Item 43*^to "no" 
by lightly crossing out the "yes" and writing "no. " 

V 

45. Ask this question directly to the teacher. 

It is very clear what the next unit should be (i. e. , rated 5) if the 
curriculum provides specific sequencing information. For example, 
the curriculum might list the following sequence: If a student(s) 
• passes Unit 1, level C, he/she should go to Unit 1, level D. If a 
student(s) does not successfully master Unit 1, level C, he/she 
should go to Unit 1, level C alternate. 
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Although the curriculum may provide little or no seqviencing informa- 
^ tion, the way in which the units'^ material are letterecvor numbered 
in itself may provide clear information. For example, it may be 
obvious that after Unit 1 is completed. Unit 2 is the unit that should 
^ be covered next. The rating in this case would also be 5. 

It is unclear what' the next unit should be (i. e. , rated 1) if the cur- 
riculum does not provide adequate sequencing information. For 
example, at the end of Unit 1, level C, the c\jrriculum might state: 
If a student(s) has not mastered the skills in this unit, review those 
skills before continuing on. 

^ It is also not clear what the next unit sBtould be if units may be 
covered in aijy sequence and no guidance is provided. 

46, Ask this question directly to the teacher. 

This question would be rated 5 (often) if the teacher rarely follows 
the sequence provided^and possibly *skips some material totally. 
The rating 1 (never) would be used if the teacher always follows 
the prescribed routing pattern or takes each unit in the order in 

which it is presented. 

t « 

47. Ask this question directly to the teache'r. * 

This question would be rated 5 (often) if the teacher constructs 
his/her own materials every day to improve sequencing and 1 if 
he/she never doeSo 

48-49. Ask Item 48 directly to the teacher. 

Record "yes'^ or "no. " If "yes, " then ask if this is the way most 
children are placed in the curriculum. If so, record "yes'' for 
Item 49 and retain "yes" for Item 48. If the answer for Item 48 
is "no, " record "no" for Item 49 and retain "no" for Item 48 or 
change the "yes" in Item 48 to "no" by lightly crossing out the 
"yes" and writing "no." 

50-51. Ask Item 50 directly to the teacher. 

If the teacher is uncertain, say, "Sometimes teachers don't always 
rely on the regular placement tests or they don't have any place- 
ment tests to begin with. Do you use some other way of deciding 
where to start the child?'* Then ask for the "specific method'* in 
order to verify the answer to Item 50. If there are no specific 
methods, record the answer to 50 as "no." If there are specific 
methods, record the answer as "yes, 
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52-53. Ask Item 52 directly to the teacher. 

Record '^es^'^ or ^^no, ^' If '^es, then ask if this is the way in 
which it is' decided that nlost children have mastered a particular 
skill. If so, -record '^es^' for Item 53 and retain '^es^^for Item 
52. if the answer for Item 53 is '*no/* record »'no" for Item 53 

• and retain ^^no" for Item 52 or change the ^^yes^^ in Item 52 to "no" 
by lightly crossing out the ''yfes" and writing "no. " ^ 

54. Ask this question directly to the teacher. ^ 

It is very clear what the next unit should be (i. e. , rated 5) if the 
curriculum provides specific sequencing information. For example, 
the curriculum might list the following sequence: If a student(s) 
passes Unit 1, level C, he/she should go to Unit 1, level D. If a 
studentj(s) does not succes sfully master Unit 1, level C, he/she 

• should go to Unit 1, level C alternate. 

Although the curriculum may provide -little or no sequencing in- 
formation, the way in which the units of material are lettered or 
numbered in itself may provide clear information. For example, 
it may be obvious that after Unit I is completed. Unit 2 is the 
unit that should be covered n^^jct. The rating in this case would 
also be 5. 

It is unclear what the. next unit should be (i. e. , rated 1) if the cur- 
'riculum does not provide adequate sequencing information. For 
example, at the end of Unit 1, level C, the curriculum rright 
state: If a student(s) has not mastered the skills in this unit, 
review those skills befo:ge continuing on. 

It is also not clear what the nfext unit should be if units may be 
covered in any sequence and no guidance is provided. 

55. tAsk this question directly to the teacher. 

This qtrg^fian would-be rated 5 (often) if the teacher rarely follows 
^ . the sequence provided and possibly skips some units of material 
totally. The rating 1 (never) would be used if the teacher always 
follows the prescribed" routing pattern or takes each unit in the 
order in which it is presented. 

56. Ask this question directly to the teacher. 

This question would be rated 5 (often) if the teacher constructs 
his/her own materials every day to improve sequencing and 1 if 
he/she never does. 

57-58. Ask these questions directly to the teacher. ^ 

Try to get at the most prevalent form of decision making. Try 
not to check all four lines. 
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59-60. Ask these questions directly to the teacher. 

6la-6lb, Ask these questions directly to the teacher. 

62-63. After reading these' questions once, explain the following: 

Self-paced means that the child's mastery of and/or interest ii) 
Specific material determines when he/she goes on to new material.^ 

If a child is not self-paced , he/she is usually in a classroom 
where progression to new material is decided by the teacher when 
the majority 'of the class has completed a unit of material. 

64-65. Item 64 is to be asked only if the teacher answered "yes" for 
Item 62. 

Item 65 is to be asked only if the tea(^her answered "yes" for 
Item 63. 

^ For both questions, ask the teacher ^Miich child has completed the 
greatest ntunber of units in the reading (or mathematics) text over 
the last two months. The teacher will most likely need to consult 
his/her records. Then ask the teacher which child has completed 
the fewest^ number of \anits in reading (mathematics) over the last 
two months. Again, the teacher will most likely consult the records.. 

Record only the number' of units,, not the child's name. 

66-67* Ask these questions directly to the teacher. Try to obtain only one 
answer for the type of grouping. K the answer is "small group, " 
ask fpr information on number, size, and re-forming of groups. 

It 

68-69. The purpose of these items ts to estimate the uniqueness and distri- 
bution of students* assignments and testing. Uniqueness means how 
many different assignments are given in a class on any one day. If 
each student redeives a different assignment, then that classroom 
would have 100% tanique assignments. On the other hand, if all 
students in a class of 25 receive the same assignment, then only, 
one 'assignment is unique and the percentage is 4%, How often stu- 
' dents are tested and how well they do on a given test give an esti- 
mate of the monitoring procedures and matching of the student and 
the curriculum. 

The date that you go into the classrom to record the information 
' must be noted in the upper right-hand corner next to "Today's 

date . " This date will be used later in coding the responses 

on the questionnaire. It must match the actual day you collect the 
information for each classroom. 
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In order to ensure that all student assignments are listed, list- 
each student in 'the ''L D. column. Using a class roster or . 
attendance sheet, li*st the initials of each child who is present 
that day. If two childreji. have the same initials, include more 
Information to distinguish' between them, such as a middle initial. 
If a student is absent on that day bu^ has been^n the classroom on 
the pr^evious day, include him/her in the I. D. list. , Students who 
are not listed should include only those who have been absent for 
two^ or more days, moved, or transferred to another class. 

In order to determine the assignment for each student, ask the 
teacher how Students are assigned classwork or homework for 
that day in reading and mathematics. Since Items 68 and 69 ^^-^ 
quire identical information, only Item 68 (reading assignments^ . 
will be discussed here. Use the same procedures, however, -fa 
complete Item 69 (mathematics assignments). 

There are several ways in which a student's assignments may be 
recorded. , If each student has an individual assignment sheet or 
prescription card, match it with the student's initials and list the 
day's ^assignment as specifically as possible. Note the name, of 
the book or text series in which the student is working, the page 
numbers assigned, and identifying information about the level or 
chapter, unit, and skill or concept. If the teacher keeps a log of 
each child's assignments, the same procedure may be' used by 
matching ^a.ch child and listiilg the assignments. If all children 
receive the same assignment for the' day, list all the identifying * 
information for that assignment and write the number of children 
counted under the I. D. column. If students are assigned work 
by groups, list the assignment and the number of children in each 
group. If all or some of the studentsOare sent to a different class 
for reading, go to that room an^, follow the same procedure for 
determining their assignment. If any of the assignments are un- 
clear or yon cannot find some of the information, use the teacher 
as a resource for clarification. Try to work quickly and cause as 
little disruption as possible. When you are finished, each student 
should be accounted for with specific assignments listed. 

If the text is only broken down into chapter,^, list other pertinent 
information such as "two-digit addition. " An example of this might, 
be: 

I^ • Book Pages Level Unit Skill 

AMS Scott-Foresman 26-31 2 ^Plurals "ing" endings 
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^ The final four columns are used to record for each child the date 

! of the last test' given, if 'that test is a diagnostic one, the number 

; of items that child had correct out of the total number of items, 

I ■ and if the score is passing. At the beginning of the school year, ' 

' you should request that teachers keep logs of this information. 

The questions- on the bottom of pages A- 10 and A- 11 will be cal- 
culated by the coder from the information that you provide on the 
forms. • ' 

Overlap 

Data. Ask the principal at the time of initial contact. Ask the teacTier 
at the time of interviewing. Include supplementary texts if the 
teacher reports using any. If the principal and teacher report 
different texts, ^ry to decide which is really being used (check 
student desks, ask students, etc.). 

70-71. NOTE; These items are intended to deterrnine those skills taught 
by the teacher that are aiso covered by the CTBS battery. These 
questions should be asked in a final interview with the teacher. 
The interview should take place immediately after the spring test- 
ing is completed or while the testing is in progress. The inter- 
viewer fcan be eithfer the same individual who conducted the pre- 
vious interviewing or ^he person administering the spring battery 
of achievement tests. ^ 
Hand the teacher tjie appropriate. form and grade level|tof the test 
battery and say: ) . 

Please circle all item nuhabers that you^think the majority of your 
students have been taught either by.thei curriculum or by.you.. Look 
at the^format of the items as well as the skills involved before de- 
ciding if an item has been taught.^ We are trying to determine if 
the test is actually, testing 'what your students have learned. For 
example, if your students, have learned to add 5+3, but this problem 
\ has always heen presented in, vertical form, you would not circle 
" that item when it appears in horizontal form.. As another example, 
if your students. have been asked questions after reading a story, 
but the questions were always taken literally from the text, then 
a question that requires making an inference from the text would 
not_ be circled. Please go through "the entire test and read the 
questions thoroughly before making a decision. 

'v When the teacher- has gone through the fentire achievement test and 
circled all appropriate items; put his/her name on the booklet with 
other identifying information such as date, school, and grade level. 
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^ In-Class Teacher Interview Questionnaire; 

Directions'for Coding 



1, Record one number only, which' is the average of the number of 
students enrolled in the fall and the number of stujdents enrolled • 
in the spring. If this average nximber is not a whole number, 
round off to the greater number of children. . ^ 

2, ^Calculate a percentage of each questionnaire (fall and spring) 
the number in Item 2 divided by the number in Item 1. .Record 
the average of the percentages over the two passes, . 

NOTE; ^'Over the two passes" refers to the fall and spring data 
' ' collection efforts, 

3, Record the average number of days a student attends, ^ 
NOTE ! On spring questionnaire only. 

4, Record a single number. This number .shouW remain constant 
from fall to spring. 

5-6. Record the total numbers. 

NOTE: On spring questionnaire only. 

7-10. Record the average over the two passes. 

11-21. Record the average ratings over the two passes. \ 

22. Assign zero (Of to lines without checks. 
Assign one (1) to lines' with checks. 

Thus, the measure consists of 5 binary codes; e.^g. , 01000 = only 
wprkbooks or wor^ksheets used in the instruction of reading. 
Combine the results over the two passes. ^ 

23. Assign zero (0) to lines without checks. ^ 

Assign one (1) to lines with checks. ^ . ' ' 

Thus, the measure consists of 7 binary* codes; e.g., 0111000 = 
flash cards, games, and manipulatives are used in the instruction 
of mathematics. Combine the results over the two passes. 

• 24. Transform all answers to hours^ consider each day to be six 
hours. Thus: 

30 minutes = . 50 

2 days = 12. 00 

3 hours = 3. 00 , . 
' Record the average over the tv/o passes. 
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25i • Assign zero (0) to lines without checks. 
Assign one (1) to lines ^ith checks. ' 

Thus', the mSfasure con|ists of 4 binary codes; e. g. , 0100 = 
only specific objectives exist. Combine the results over the 
two passes. ^ . * 

26. Record the average rating over the two passes; record zero (0) 
if no* specific objectives exist. 

27. Assign zero (O) to lines without ch^ck^. * 
Assign one (1) to lines with cihecj^. ^ 

. Combine the results over the two passes. 

28. Record the average rating over the two passes. Record zero (0) 
if there are no objectives. 

29. Assign zero (0) to lines without checks. , 
Assign one (1) to lines with checks. 
NOTE ; On fall questionnaire only. 

30. Assign zero (0) to lines without checks. 
Atesign one (1) to lines with checkg. 
Combine the results over the two passes. 

An example of combined results: In the fall, a new reading objec- 
tive is introduced once a week (code = 00100) and in. the spring, 
9. new reading objective, is introduced olDce a month (code = 00001). 
The combined results are coded 00101. 

31. Record trie ajverage amoxmt of time per week over the two passes. 

32. Assign zero (OTHo lines without checks. 
Assign one (1) to lines with checks. 

Thus, this measure consists of 4 binary codes; e.g., 0100 = only 
specific objectives exist. Combine the results ov^r the two passes. 

33. Record the average rating over the two passes. ^-^""^^ 

34. ' Assign zero (0) to lines without checks. 

Assign one (1) to lines with checks^ - 
Combine the results over the two passes, 

35. . Record the average rating over -the two passes. 

36. Assign zero (0) to lines without checks, 

' Assign one (l)-to lines with checks. "9 
NOTE ; On fall questipnnaire only. 

37. Assign zero (0) to lines without checks. 
• Assign one (1) to lines with checks. 

Combine the results over* the two passes, . 
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38. Record average amouht of time per week over the two passes. 

39. no 0 ^ . . 

yes = 1 A 

Record one (1) if "yes^' on either fall or spring pass. Record 
zero (0) if ^^no^^ on both passes. 

40. No code--not recorded. 

41. no = 0 ' . 
yes = 1 

Record one (1) if ''y^s'^ on either fall or spring pass. Record 
zero (0) if "no'' on both passes. 

42. No code--not recorded* 

43. no = 0 ' 

yes = 1 

Record one (1) if ''yes" on either fall or spring pass. Record 
zero (0) if "no" on both passes. 

44. No code--not recorded^ 

45-47. Record the average ratings over the two passes. 

^ 48. no =^ 0 
yes = 1 

Record one (1) if "yes" on either fall or spring pass. Record 
zero (Q) if '^no'^ on both passes. 

49. No code--not recorded. \ 

50. no = 0 

yes = 1 ' 

Record one (1) if "yes" on either fall or spring pass. Record 
zero (0) if "no" on both passes. 

^51. No code--not recorded. ^ 

\ 

52. no = 0 , 
yes = 1 M 

Record one (1) if "yes" on either fall or spring pass. Record 
zero (6) if "no" on both passes. 

53. No code--not recorded. 

54-5?), Record the average ratings over the two passes. 
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57-60. Assign zero (0) to lines without checks. 
Assign one (1) to lines with checks. 
Combine the results over the two passes. 

6la-6lb. There are several parts to analyzing the answers to these ques- 
tions. After the fall interview, one coder should make a master 
' list of all uni(|ue answers. This list is then. used to create another 
list of no more than five major categories of responses. Code 
each questionnaire using I (yes) or 0 (no) for each category. 

Example: Suppose the five major categories generated from all 
the questionnaires are: tutor, special work, repeat material, 
continue on, give homework. If a particular teacher says he/she 
only tutors students who are not progressing toward mastery, 
you would code his/her answer as 10000. 

The major categories generated in the fall should be used again 
in the spring. So, if the same teacher reports in the spring. that 
he/she gives homework, his/her answer would be coded 00001. 

If the results are the same^over the two passes, use them; if 
different,^ record the combined results (e. g. , the combined re- 
sults of the above answers would be 10001). 

62-63. no ^ 0 . 

yes = 1 » 

Record one (1) if »^yes'» on either fall or spring pass. - Record 
zero (0) if ^*ho** on both passes. 

64-65. Take the largest number and subtract the smallest number to 

calculate the answer. Average the answers over the two passes. 

66-67. Assign zero (0) to lines without checks. 
Assign one (1) to lines with checks. - 
Combine the results over the two passes. 

Record the average over the two passes for the number of 
groups and group sizejr. 

Transform the answer to the regrouping question into frequency 
by mqnlh (e.g. , ''once a week=4, once every 2 months=.5). 

68-69. There are four responses that must be coded for each subject area, 
i. e. , reading and mathematics. They are: percent of unique 
assignments, average number of days since the last test, range 
of days since the last test, and average percent correct on the 

• . last test. The procedures for coding reading and mathematics 

are the same; only reading will be used for explanatory purposes. 
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The fir-st item {68a) is determined from the assignment identifica- 
t^on information: book or text series, assignment pages, level, 
vinit, and skill. Look at the first assignment. Put a 1 next to it. 
Look at the rest of the assignments and check off { /) any other 
assignment that is the same as number 1. Do not check off num- 
ber 1. Look at the next assignment on the list that is not dhecked 
off. Pyit a 2 next to it. Look at the rest'of the assignments and 
check off any that a^e the same as number 2. Continue in the same 
manner to number 3 ajid so on through all the assignments until 
each>assignment. is either numbered or checked. When you are 
finished, be svire that^ all of the numbered assignments are unique. 



1. e. 











Pages 


Level 




Skill 


2^-31 


B 


2 " 


9 


40-43 


C 


1 


6 


10-11 


B 


3 


1 


'40-43 


c ■ 


1 


6 



Example: 

Book 

1. Scott -Foresman 

2. Scott-Foresman 

3. S-F Workbook 
/ Scott-Foresman 

In this case, three assignments are unique. Using the last num- 
bered assignment, divide by the total number of assignments listed 
to get the percent of unique assignments. In the above example, 
3 out of 4 assignments- are unique; therefore, 75% would be the 
response to Item 68a, ■ 

The column "Last Test Given" will supply the response to Items 
68b and 68c. For each date, count the nvunber of school days 
(excluding weekends, holidays, in-service days, etc.) from that 
date to the date noted in the upper right-hand corner (Today's Date), 
counting Today's Date as 0. Put the number of days since the last 
test next to the date of the last test. 



Example ; 



Today's Date 10/4/76 



Last Test Given • 

9/24/76 (5) 

10/1/76 (1) 

9/13/76 (15) 

10/4/76 (0) 
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To determine the average- number of days since the last test (68b), 
stun the number of days and divide by the total number of assign- 
ments listed. In the example shown previously, the sum of the 
days is 21; therefore, the average number of days since the last 
test would be 21/4 - 5.25. To determine the ranj^e of days (68c), 
find the greatest nuinber of days since the last test and subtract 
the least "humber of days since the last test. In the above example, 
the range would be 15-0 or 15. " 

The column labeled "How Many Items Did the" Student Get Correct 
out of the Total" is used to determine the average percent correct * 
on th,e last test (68d). For each scor^, you must determine the 
percent correct. In other words, if a student scores 6 out of 10 
possible points, he/she has scored 60% correct. When you have 
found the percent correct for each student to two decimal places, 
sum the percentages and divide by the total number of students. 

Example: 



How Many Items Did the 


•Percent 


Student Get Correct out 


. of the Total 


Correct 


6/10 


60.00 


8/15 


53.33 


9/9 


100.00 


21/25 


84.00 


7/16 


■ 43.J5 


1 


E=341.08 



If the sum of the percentages is 341. 08> divide by 5 (the number of 
assignments listed). Then the average percent correct on the last 
test would be 68.22%. . ' ; 

The information on diagnostic testing will be used- in a behavioral 
analysis of the curriculum (see Appendix B-Z). ; • ^ 

Repeat all of the above steps for Item 69 using Ihe assignments 
listed in mathematics. 

Overlap 

Data. Questions about the text or series being used will not be coded. 

This information will be used by the curriculum expert to answer 
Items I and 2 on the Curriculum Analysis Questionnaire. 
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70-71. Coiint the njimber of items circled in all reading tests. Reading ^ 
iests for gradie 1 include: 

Letter Soxrnds 
. Word Recognition I and II 
Rea>ding Comprehension 
Language I 

Reading tests for grade 3 include: 

Readings Vocabulary 
Reading Comprehension 
Language Expression 

Trtie sum of all circled items in reading is then divided by the 
total nximber of it-^ms to determine the percentage of overlap. 
The total number of reading items for^ grade 1 is 100 . There- 
fore, if the s-um of all circled items is 73, then the teacher's 
estimated percentage of reading overlap is 73%. 

The total number of reading items for grade 3 is 115 .. If the sum 
of all circled items is 81, then the overlap estimate is 70.43%. 

Count the number of items circled in all math testSo Math tests 
for grades 1 and 3 include: 

Mathematics Concepts and Applications 
^ Mathematics Computation 

The* achievement battery for grade 1 includes a total of 56^ items in 
mathematics. Th^ battery for grade 3 contains 98 mathematics 

items. ' 

* ■ , ' 

Sum the number 6f circled items and divide by the total number of 
items to determine the percentage of overlap in mathematics. For 
*e;xample, if a first-grSde teacher circled 40 items on the math tests, 
divide 40 by 56, which yields a percentage of 71.43%. 
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A^NALYZING CURRICULA 



Page 



Curriculum Analysis Questionnaire B-1 

^Curriculum Analysis Questionnaire: 

Directions for Obtaining Measures B-6 

' Curriculum Analysis Questionnaire: 

' Directions for Coding B-17 

Analyses of the Reading and Mathematics Tests 
^ of the CTBS (Expanded Eldition), Grade 1, 

Level B, Form S . B-19 

Analyses of the Reading and Mathematics Tests 
of the CTBS (Expanded Edition), Grade 3, 
Level 1, Form T, Book 1 B-31 



Note : Appendix B-2 begins on page B-5 3, 
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Curriculum Analysis Questionnaire 

Name of Curricvilum Expert 

Date 

School District ^ 

School ■ 

Grade * 

Teacher's Name 

1. What is the average percentage of 
, overlap in mathematics ? 

2. What is the average percentage 

' of overlap in reading? 

3. Rate the reading materials generally available in the classroom 
as to how interesting the ^tudents find them. 

Not Very 
Interesting Interesting 

1 2 3 4 5 ^ 

4. Rate the mathematics materials generally available in the class- 
room as to how interesting the students find them. 

Not . . Very 

Interesting Interesting 
I 2 ^ 3 , 4 5 

5. Withirx a single mode of reading instruction such as pencil and 
paper, rate the variety of instruction format. 

No Large Amo\mt 

Variation " ' . of Variation 

-1 2.3 4 5 

6. Within a single mode of mathematics instruction such as pencil 
and paper^ rate the variety of instruction format. 

No " Large Am<kint 

Variation of Variation 

1 2 3 4 5 
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What types of objectives are present in the reading curriculum? 
(check one) 

General 

Specific 



Both general and specific 
No objectives at aril 



If any specific objectives are present, rate the reading curriculum 
on the clarity of its specific objectives, 

TT 1 , Clear 

Unclear * . . • r- 

1 2 ^ ^3 4 5 

When a new objective is presented -in the reading materials, ^ 
which of the following appear ? , (check appropriate lines) 

A written statement of the objective 

An example of the new objective 



A range of examples of the new objective 

Practice problems for the student to complete that 

, are different from the example ^ 
A no n- example 

Rate the reading curriculum as to how closely the available 
materials match the stated objectives. 

Closed 
Match 

Match ^ - 

1 2-3 4 5 

Which of the following assessment procedures does the reading 
currictilum include? 

Specific Method 



Placement procedure _____ r 

Monitoring procedure _ — . — , 

Method for assessing mastery 

of skills : , 

Rate the reading curriculum as to the degree to which the student 
and the curriculum can be easily matched. 

^. rr. 1. Easy to 

Difficult , : , 

, -Kjf 2. X. Match 
to Match ^ ^ 

1 2 3 4 5 
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13. For reading, rate the degree to which it^ms on the placement 
test reflect the curriculvim content. 

No High 
Relation ^ Relation 

1 2*3 4 5 

14. For reading,' r%te the degree to which items oh the mastery test 
reflect the curriculxim content. ' 

No High 
Relation . . , . Relation 

J 2 .3 4-5 

15. What types of objectives are present in the mathematics c\ir- 
riculum? (check one) 

General 

Specific 



Both general and specific 
No objectives at all 



16. If any specific objectives are present, rate the mathematics cur- 
riculimi on the clarity of its specific objectives. 

Unclear Clear 
1 2 3 4 5 

17. When a new objective is presented in the mathematics materials, 
which of the following appear? (check appropriate lilies) 

A written statement of the objective 

An example of the new objective 



. A range of examples of the new objective 

Practice problems for the student to complete 

that are different from the example 

A non- example 

18. Rate the mathematics c^rijiculum as to how closefer the available 
materials match the stated objectives. 

No . Close 

Match Match 



1 



5 
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19. Which of the following assessment procedures does the mathematics 
curriculum include: 

Specific Method 

Placement procedure ^ 

Monitoring procedure ^ 

Method for assessing mastery 

of skills 

20/ Rate the mathematics curriculum as to the degree to which the 
student and the curriculum can be easily matched. 

Difficult Easy to 

^ to Match Match 
1 ' 2 3 4 5 

21, For mathematics, rate the degree to which items on the placement 
test reflect the curriculiim content. 

No High 
Relatioh. " Relation 

1 2 .34 5 

22. For mathematics, rate the degree t-o which items on the mastery 
, test reflect the curriculum content. 

No . High 

Relation ' Relatio^i 

1 ' 2 3 4 5 

For the following set of questions, please respond to each question for 
reading and for mathematics. 



Reading Mathematics 



23a. Is there a systematic way of assess- 
ing student initial abilities built into 
the curriculum (e.g., are some types 
of placement tests provided) ? 

*23b. Pleas§;* specify the method of assess- 
ing student initial abilities. 

24a, Is there a systematic way of assess- 
ing student mastery of specific skills 
built into the curriculum (e. g. , are 
tests provided) ? 

24b. Please specify the method of assess- 
ing student mastery. 
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Rate the reading curriculum according to how clear it is what the^ 
next unit shotild be at the completion of a unit of material. 

Very 

Unclear Clear 
1 2 3 4 5 

Ra%e the mathematics curriculvim according to how clear it is .what 
the next unit should be at the completion of a unitrof material. 

Very 

Unclear ' ^1^^^ 

1 2 3 4 ' 5 

If a student does not pass a test, what does the reading curriculum 
suggest doing? 

Tutor 

Give special work ^ ^ 

Continue on [ 

Give extra homework 



Repeat the material covered 
Other, please specify 



K a student does not pass a test, what does the mathematics cur- 
riculum suggest doing? 

Tutor 

Give* special work 

Continue on 

Give extra homework . • . 
Repeat the material covered 

Other, please specify . ^ 



What is the principal sequencing system in use in the reading 
curriculum? 

Unordered 

Linear 

Branched ^ * * 

Modular 



What is the principal sequencing system in use in the mathematics 
curriculum? 

Unordered ' 

Linear " , 

Branched 

Modular 
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Curriculum Analysis Questionnaire; ^ 
Directions for Obtaining Measures 

1-2. The purpose of these measures is to assess the degree to which a 

particular curriculum and an achievement test overlap. In order ^ 
to estimate these measures, match the test to the curriculum by 
finding the percentage of the items on the test that are covered by 
the curriculum. When this information is combined with the amount 
of the curriculum covered during the school year, an estimate can 
be made of the opportunity that a group of students has to learn the 
information tested on the criterion. 

Before trying to match an ^achievement test with a curriculum, 
scan both in order to gain some familiarity with the material. If 
a teacher^s annptated edition of the text is available, look at such 
things as the table of contents, goals and objectives, and general 
layout of the work presented. Also look at the publisher's scope 
and sequence chart if it is available. For the achievement test 
being used, look at the technical manual item descriptors, the 
subtests, the kinds of items included, the format of tile items, 
and the directions for administering the test. 

A listing of the item types and fqrmats used in the CTBS is presented 
later in this ^Appendix. This listing includes information concerning 
the skill or concept being tapped (e. g. , addition facts less than 10) 
as well as the form of presentation (e. g. , word problem, horizontal).. 
Using it as a guide, begin to move through the curriculum. It is 
probably easiest to start at the beginning of the text, that is, with 
• Chapter 1, *Unit 1, or similar subdivisions of the curr iculum. As 
each item is located in the curriculum materials, mark the chapter 
or unit in which the item is first covered (see Example 1, page B-7). 
• The item may be covered later as review at practi<;e; however, 
only the initial appearance of that item should be listed. After you 
complc^te'^the first unit of the curriculum, §um the ntii^lpet of items 
coy^ed in that unit and calculate the percentage of the tqtal number 
ofltems in the test'. An example for a test containing a total of 
100 items is: 

Unit Items CoV^ered \ % of Total 

. Chapter 1 6 ' p 6% 

Work through each unit in the same fashion;by mar^iing the unit or 
chapter in which a test item is covered and by noting the number 
and percentage of total test items covered within that unit. 
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The following is an example of a listed item format ajtid how it 
may be covered in the curriculum: ' 

Item Format: 3 addends, sum < 10, horizontal 
Curriculum Content: 

0 

00 . 00 ' 0 

These are John^s. These are Jeff^s. This is Karl^s. 

How many are, there in all? 

This is the number sentence we would use: 

2 + 3 + 1 + 



Tljis material was found in Chapter 2, along with other problem^ 
of a similar type. Therefore, next to the item format, write 
^'Chapter 2, " At the end of Chapter 2, note that 5 test items were 
covered in that chapter. If the test contains 100 items, this would 
be 5% of the total items: 

Unit Items Covered % of Total 

Chapter 2 ' 5 5% . 

The ^percentage of overlap between a particular curriculum aild 
an achievement test is estimated by the cumul'ative percentage 
that should be calculated by adding each new unit's percentages. to 
the previous percentage (see Example 2, page B-9)« per- 
centage of overlap must be calculated for each curaridulum (read- 
ing and mathematics) for each classroom. One number must be 
recorded for each curriculum for each classroom according to 
the following rules, . Example 3 (page B-9) is provided to aid you 
in these calculations, * 

1, If the entire class is using the same single series 
and the entire class ends at a single spot in the 
curriculum, use the cumulative percentage of 
overlap of that spot. 



When asstiming that a test item or test item content is covered in a. 
'- text series, care should be taken to ensure that the mann^ of pre- 
sentation is similar with respect to the availability of cues for the 
item's solution. For example, in the item set given above, if the 
student were always exposed to both the pictures and the equation, 
but the test only gave the equation, one could not assume the informa- 
tion had been taught. 
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Example 2 



Unit 




Items Covered 


% of Total 


Cumulative % 


Chapter 


1 


6* 


6 


6 


Chapter 


2 


,• 5 


5 


11 ^ 


Chapter 


3 


9 


9 


* 20 



Example 3 



Name of Student Curriculum Series Used 



Final Curriculum Loca- 
tion (Page Number, Unit, 
Level, Skill Number, etc. ) 



Aloysius Alhazen 



Abracadabra 



48, J, 10 



n 
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5-6. 



7. 



in (2). -"A^Aviauais, proceed as 

students would find a pZucul^! " interesting 

^hat you have practical i^o"!:,^^^^^^^^ We don't expect 

much the children are intereTted^" ' " ^'""'"'"^ 

contained i^ the text. WeZZ\ t the material 

qualities of the maferiaTs flo^ ""Z Motivational 

i- e. , do the mater als apptr^ theoretical point of view. 
Please rate the curriculn^ ^trinsically interesting ? 

si^ering ^.ch thin^T aT^^i^TLr^^^ - ^on- 

tion such as textbook or Workbook and "^""^.^^^^^^^ of presenta- 
occurs within that mode. Cons^ de'r variar " ' '""^ ^^^^ 
sentation, layout, and type of "^^^'^^^^^ patterns of pre- 

fer example, a progra^^thal^r L^rt^xtTlf '^.^'^ 
for the majority of instruction has Si questions 
-ted 1. On .the other hand, a progrim tlatT"" 
^ariety of activities, such as wrSrsho 't a" 
plays, and writing questions for othe/s udl r"'"' ""'^^^^ 
Have a large variation within that tZj^fllll ITZl', ^^^^ 
A general objective is a global, program goal. 
Examples of general, reading objectives: 

1. The child will become an independent reader 

-hoosxng to read for pleasure. 
^. The chHd will be able to extrarf rv^o. • r 

written text. extract meaning from 
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A specific objective is one that is linked with the instructional 
materials and is most' often stated in behavioral terms. 

Examples of specific reading objectives: 

1. The child will be able to decode all one -syllable 
words in his /her reading text.. 
^ 2. The child will be able to read a short paragraph 

story (three or four sentences) and paraphrase the 
story. 

' 8. If no- specific objectives are present in the reading cur.riculum, 
skip to Item 9. If there are some specific objectives in reading, 
answer this question: i > 

Consider the clarity of the specific objectives in the reading cur- 
riculum. On a scale 'from 1 to 5, ^fate the clarity of the specific 
objective 6. 

A clear objective is one that is focused and narrow. It precisely 
states what the child will know how to do on completion of the 
materials. An example of a specific objective that is clear (i. e. , 
rated 5) is: upon completing this unit, the student will be able 
to pronounce all one-syllable words of the CVC formi 

An unclear objective is one that is dispersed and broad. It does 
not describe precisely what the child will be able to do on com- 
pletion of the materials. An example of a specific objective that 
is unclear (i.e. , rated 1) is: upon completing this \init, the child 
knows how to read one-syllfitble words. 

i 

9, A written statement of a reading objective may appear either in the 
teacher's manual or the child's text or in both. An example of a 
written statement of a reading objective is: the child will be able 
to pronounce any one-syllable words with two vowels, the final 
vowel being a silent "e. " 

An example of the new objective is: to present the child with a 
list of words that conform to this rule (sale, bale, male) and to 
pronounce the words for him/her. 

A range of examples covers the broadest extremes of the domain. 
An example is: sale, Joe, ate. 

Practice problems for the objective could be a list of words con- 
fo3:ming to the rule that the child must pronotince by hiriiself /her- 
self. 

A non- example for the objective is one outside of the domain. 
An example is: rut, not, near. 
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10. The materials match closelyv (5) if the contents of both the materials 
and the objectives are identical. The materials do not match the 
objectives (1) if the contents afe not identical. j 

11. A placement procedure is a systematic method (either overall 
placement or irnit pretest) built into the curriculum for assessing 
student initial abilities. It may be a diagnostic test, a student-r 
teacher .interview, a conference with the previous year 's teacher, 
etc. , , 

A monitoring procedure is a way (e. g. , test, conference) of diag- *' 
nosing which new material is prescribed. The procedure takes | 
place prior to any^ attempt to teach materials. 

A method for assessing mastery monitors how well a section of 
material has been learned. It may be a test, conference, etc. , 
and is given after the child has studied the material. 

If the reading curriciolum has any of these procedures, please 
specify tha method, such as test, conference, etc. 

12. The reading curriculum and the student would be easy to match 
if there are a placement procedure built into the curriculum, a 
moijiitbring procedure for prescribing new materials, and a method 
for ^assessing mastery of studied material. 

It, would be difficult to match the reading curriculum and the stu- 
dent if all three of the above procedure were missing. 

13. This question is- very much like the overlap question (Item 2). It 
is concerned with the degree to which the placement tests that' are 

^ built into the reading curriculum contain the same type of informa- 
tion as the curriculum. Unlike the overlap question, however, this 
question does not ask that you estimate the degree of overlap through 
a specific procedure. We ask that you make a judgment oh the basis 
of your familiarity with the curriculum you are evaluating. ^No 
relation (1) between placement tests and the curriculum coitetit 
means: (1) there are no placement tests built into -the curriculum, 
or (2) the placement tests are in no way related to the curriculxim « 
content (e*. g. , a test for gross motor skills fs used to place 
children in a reading group). High relation (5) means that for 
e^ch item on the placement test there is a matching item in the 
cxifticulum content. 

14. This question is the same type 6f question as Item 13." Ybu are 
asked to judge the degree of overlap between the curriculum con- 
tent in reading and mastery tests . 
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15. A general objective is a global, program goal. 

Examples of general mathematics objectives: ^ I 

1. The child will demonstrate problem-solving skills. 

2. The child will enjoy playing mathematical ganies. , 

A specific objective is ofte that i^ linked with the instructional 
mateiriala and is most often stated in behavioral terms. 

Examples of specific mathematics objectives: 

1. The chil4 will be able to add two one -digit numbers. 

2. The child will be able to count to 100 by fives. 

16. If no specific objectives are present in the mathematics curriculum, 
skip to Item 17. If there are some sp^ific objectives in mathe- 
m^ics, answer this question: 

As you did for reading, consider for mathematics the clarity of 
the specific objectives. On a scale from 1 to 5, rate the clarity 
of the specific objectives. 

A clear specific objective that would be rated 5 might be: given a 
series of one-digit two-numeral addition problems in the vertical 
form, the child ^ill be able to'^correctly ca^lculate their sums;,. 

An unclear specific objective (1) might be: given a set of one-digit 
addition problems, the child will understand how to find their sxims. 

17. A written statement of a mathematics objective may appear either 
in the teacher's manual or the child's text or in both. An example 
of a written statement of a mathematics objective is- the student 

^ will be able to sum any two one-digit whole numbers. 

An example of the*new objective is: to present the child with a 
list of solve^ problems and conceptual methods for calculating 
correct answers to the problems (e.g. , number line representa- 
tions of the solutions to 3+1=4, 4+1=5, 1+2=3), 

A range of examples covers the broadest extremes of the domain. 
An example is: 5+4=9, 8+9 = 17, 6+7=13. 

Practice problems for the objective cotild simply be a list of 
unsolved probleiJ^ *for the student to complete. 

A non- example for the oTDjective is one that is outside of the domain. 
An example is: 1-1 = ? 

18. ' The materials match closely (5) if the contents of both the rnaterials 

and the objectives are identical. The materials do not match the 
objectives (1) if the contents are not identical. 
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19, A placement procedure is a systematic method built into the cur-c 
riculum for assessing student initial abilities. /It may be a diags 

tnostic test, a student -teacher int'erviej^w, a confisrence with the . 
previous year's teacher, etc. 

A monitoring procedure is a way (e. g. , tef t, conference) of diag-* 
nosing which new material is prescribed. The procedure takes' ' 
place prior to any attempt to teach materials. 

A method for assessing mastery monitors how well a se^ij^ion of 
material has been learned./ It may be a test, conference, etc. , 
and is given after the child\has studied the material. 

If the mathematics curriculum has any of these procedures, 
please specify the method, such as test, conference, etc. 

20, The ""mathematics curriculum and the student would be easy to 
match if there are a placement procedure built, into the curriculum,* 
a monitoring procedure for prescribing new materials, and a 

^method for assessing mastery of studied material. 

It would be difficult to match the mathematics/curriculum and 
the student if all three of the above procedure were missing. 



21, This question is ver^y much like tfle overlap question (Item 1). 

It is concerned witli the degree to which the placement tests that 
are built into the liiatjiematics curriculum contain the same type 
of information as the curriculum. Unlike the overlap question, 
however, this question does not ask that you estimate the degree 
of overlap through a specific procedure. We ask that you make 
a judgment on the basis of your familiarity with the curriculum 
you are evaluating. No relation (1) between placement tests and 
the curriculum content means: (1) there are no placement tests 
built into the curriculum, or (2) the placement tests ar'e in no way 
related to the curriculum (e.g. , a test of color recognition is used 
to place children in mathematics). High relation (5) means that 
for each item on the placement ?est there is a'matching item in 
the curriculum content. 

22. This question is the same type of question as Item 21. You are 
asked to judge the degree of overlap between the curricxilum con- 
tent in mathematics and mastery tests. 
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23a-23b. A systematic method for assessing student initial abilities must 

be built into the curriculum for this questidn to be answered "yes. 
It may be a placement teat, a student-teacher interview, a con- 
ference with the previous year's teacher, etc. It can be either an 
overall placement assessment, or an assessment at the beginning 
of each unit. If no such method exists, answer "no. " 

If a specific method is present, please specify it. 

24a-24b. A systematic method for assessing student mastery must be built 

into the curriculum for this question to be answered "yes. " It may 
be a test, conference between student and teacher, etc. The assess- 
ment of what the student has learned must be made after he/she 
has studied the material. If no such method exists, answer "no.^' 

If a specific method is present, please specify it. 

25-26. It is very clear (5) what the next unit should be if the curriculum 
provides specific seqiiencing information. For example, the cur- 
riculum might list the following sequence: If a student(s) passes 
Unit 1, level C, he/she should go to Unit 1, l^vel D. If a student(s) 
does not successfully master Unit 1, level ^, he/she should go to 
Unit 1, level C alternate. 

On the other hand, the curriculum may provide little or no sequencing 
^ information but the way in which the units of material are lettered or 
numbered in it^self may provide clear information. For example, 
it may be obvious that after Unit 1 is completed, Unit 2 is the 
unit that should be covered next. In this case, the rating would 
also be 5. , 

It is unclear what the next unit should be (i. e. , rated 1) if the cur- 
riculum does not provide adequate sequencing information. For 
' • example, at the end of Unit 1, level C, the curriculum may state: 

If a.itudent(s) has not mastered the skills in this unit, review those , 
skills before continuing on. 

It is also unclear what the next unit should be if units may be covered 
in any sequence and no guidance is provided. 

27-28. Check those items that correspond to the suggestions in the text.. 

29-30. An unordered sequencing system- is one in which students proceed 
through units in any order. 

A linear sequencing system is one in which there is a specified 
progression of units. No choice is given for varying this pro- 
gression. 
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A branched sequencing system is one in which there are alterna- 
tive paths available to students after completing each vinit. When 
an alternative unit is chosen, there iexists a new series of alterna- 
tive units through which to progress. 

^There are more alternative routes in a modular system than in a 
branched sequencing system. 
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Curriculum Analysis Questionnaire; 
Directions for Coding 



1-2. IJecord the percentage of overlap (cumulative percentage) for 
each classroom according to the rules that were stated on the 
''Directions for. Obtaining Measures. '* 

' '3-6, Record the niimber of the rating directly. 

7. Assign zero (0) to lines without checks. 
' Assign one (1) to lines with checks. 

Thus, the measure consists of 4 binary codes, e. g. , 
01000 = only specific objectives exist. 

8i Record the number of the rating directly. * 

9. Assign zero (0) to lines without checks. 
Assign one (1) to lines with checks. 

10., Record^t^e^umber of the rating directly. . 

11. To co^ie the first €bltTmn, assign zero (0) to lines 

- without checks and assign one (1) to lines with checks. 

To code the ^^specific method" column, several steps are required 
After the questionnaires are completed, make a list of all unique 
answers in this column. Many of these uniquje answers will be 
similar and can be clustered into the same category. This list 
of unique methods should then be used by one coder to create a 
master list of no more than five major categories of response. 
Using this master list, code the answers in the method column' 
using 1 (yes) .and 0 (no) for each category. Example: Suppose 
the unique answers in this column cluster into only three major 
categories: testing, conference with student, and discussion 
with teacher of previous school year. If the specific method for 
placement in a particular questionnaire is testing', then the code ' 
would be 100.. • * 

Follow this procedure for placement, monitoring, and mastery 
separately. , - 

12-14, Record lahe number of the rating directly. 

15. Assign zero (0) to lines without checks. 
Assign one (1) to lines with checks. 

16. Record the number of the rating directly. 

17. Assign zero (0) to lines without checks. 
Assign one (1) to lines with checks. 
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18. Record the number of the rating directly, 

19. To code the first column, ^tssign zero (0) to lines without 
checks and assign p'ne (1) to lines with checks. 

To code the '^specific method" column, several steps are re- 
quired* After the questionnaires are completed, make a list 
of all unique answers in this column. Many of the unique answers 
will be similar and can be clustered into the same category. This 
list of unique methods should then be used by one coder to create 
a master list of no more than five major categories of response. 
Using this master list, code the answers in the method column 
using 1 (yes) or 0 (no) for each category. Example: Suppose the 
unique answers in this column cluster into only three major 
categories: testing, conference with student, and discussion 
with teacher of previous school year. If the specific method 
for placement in a particular questionnaire is testing, then the 
code would be 100, 

Follow this procedure for placement, monitoring, and mastery 
separately. 

20-22. Record the number of the rating directly. 
23a-24a. For each curricul\im (mathematics and reading), record: 
no 

yes = 1 

23b-24b. After the questioimaires are completed, one coder should make a 
master list of all unique answers. Use the master list to create 
no more than five major categories of responses. Then code 
each answer using 1 (yes) and.O (no). Do this for each curriculum. 

25-26, Record the n\imber of the rating directly, * 

27-30. Assign zero (0) to lines without checks. 
Assign one (1) to lines with checks. 
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APPENDIX B -2 
BEHAVIORAL ANALYSIS OF CURRICULA" 

Page 

Behavioral Analysis of Curricula Data Sheet B-53 

Behavioral Analysis of Curricula Data Sheet: 

Instructions B-57 

Behavioral Analysis of Curricula Codin'g Sheet B-67 

Additional Information B-68 



The measures included in this Appendix are designed to assess the 
quality of individualized instructional materials in both mathematics 
and reading. The measures require the presence of diagnostic tests 
and procedures for individual decision making in instruction. There- 
fore, they cannot be estimated for all curricula in the study. It is 
recommended that they be estimated whenever feasible and that sep- 
arate analyses be performed on classrooms for which these measures 
are obtained. 
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Behavioral Analysis of Curricula Data Sheet 

Name of Curriculurn Expert • 

Date 

School District 

School 

Grade 

Teacher's Name ^ 

Measures of Adapting Variables 

1. Consequence ratiO i>. Using the in-class interview. 
Items 68 and 69, identify the most recent diag- 
nostic test taken by each child. Locate each test 
in your copy of the curriculum materials and 
identify an appropriate "unit'* to be counted and 
compared across classrooms. This unit must. ■ 
be constant within classrooms. Count the num-. 

ber of units in each diagnostic test and add them. 1. 

2. Using the teacher's edition of the curriculum 
material as your guide, locate all the material 
the developer suggests be prescribed when diag- 
nostic test performance does not indicate mas- 
tery. Count the units in the developed 's recom- 
mended consequent material for each diagnostic 

test. Enter this number. 2, 

3. Using the in-class interview. Items 68 and 69, 
identify the teacher 's prescription "for each diag- 
nostic test failure. Locate this prescription in 
your copy of the materials and compare, in terms 
of number of arbitrary units, the developer's rec- 
ommended prescription. Note the amount and 
direction of each deviation, and enter a total cor- 
rection number (with plus or minus sign) for "the 

. teacher^s modification of the developer's pre- 
scription. 3. 
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4. Using the in-class interview. Items 68 and 69, 
check for any teacher prescription of material 
consequent to a diagnostic test pass . Since the 
developer ordinarily recommends that teachiiig 
material be skipped consequent to a pass, the 
teacher who prescribes material consequent to 

a pass is modifying the actual consequence time. 
Count the number of units in all material pre- 
scribed consequent to a pass and enter that 
number. . \ 4"; 

5. To adjust the developer's recommended con- 
sequence for actual classroom use, a;dd or sub- 
tract (depending on sign) the number entered 

on line 3 to /from the number entered on line 2, 5. 

6. As a second adjustment of the developer's recom- 
mended consequence for actual classroom use, 
subtract the number of \mits entered on line 4 

from the number entered on line 5. 6, 

7. Add lines '1 and 6 to find the "time, " i.e. , total 
number of \mits, used in both diagnosing and 
teaching. 7. 

8. The consequence ratio is consequence "time" 
divided by total "time. " Divide the number 
entered on line 6 by the number entered on 
line 7. Take this division to two significant 
places and enter that number on line 11 and 
on the coding sheet (page B-67). This is the 
consequence ratio. 8. 

9. Discriminability ratio. Using data from the 
same in- class interview question, count 

the number of students who passed a diagnostic 

test. 9. 

10. Using data from the same in- class interview 
question, count the number of students who 

failed a diagnostic test. 10. 

11. The discriminability ratio is the number of 
diagnostic test passes failures (whichever 
is smaller) divided by the total number of stu- 
dents in the class. Select the smaller number 
of lines 9 and 10 and divide this smaller num- 
ber by the number of students in the class, 
counting students who did not take a diagnostic 

test. Take this to two significant places. 11. 



ERIC 



184 



I 



B-55 



12. Double the number entered on line 11 and enter 
on line 12 and on the coding sheet. This is the 
discriminability ratio. ^12. 

13. Predictive validity ratio. ' Using data from the 
in-class interview, Items 68 and 69, covint the 
number of students whose diagnostic test per- 
formance indicates a '^skip^' and who also passed 

' the mastery test. Enter this number. 13. 

14. Using data from the in-class interview, Items 
68 and 69, count the number of students whose 
diagnostic test performance indicated a "take" 
and who also failed the mastery test. Enter 
this number. 

15. Add the entries for lines 13 and 14 to find the 
number of correct predictions. 15. 

16. Using data from the same in-class interview 
question, count the total number of students 
for whom a diagnostic test decision was made. 

Enter this number. 1^* 

17. The predictive validity ratio is the number 
of correct predictions divided by the number 
of decisions. Divide the entry on line 15 by 
the entry on line 16. Take this division to two 
significant places and enter this number on 
line 17 and on the coding sheet. This is the 
predictive validity ratio. 1'7* 

18. Select the lowest number of lines 8, 12, and 
17. Enter this lowest number on line 18 and 

on the coding sheet. 18. 



Measure of Appropriately Determined Response Opportunities 

19. For each child* s assignment in reading class, 

determine the number'of response opportunities. 
Sum these individual totals to fiiad a classroom 
total of response opporttinities . Enter this class- 
room total, 19» 



Because this measure reqtaires classroom intervention that may be diffi- 
cult for the contractor to accomplish, it shoxild be considered an optional 
measure. 
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For each response opportimity, answer »'yes" 

or ^'no^' to the following question: '*Can a child 

respond correctly without behaving in the manner 

described by the stated or inferred objective?'^ 

Sum the number of "yeses'* to this question and 

enter this number. 20. 

21. For those response opportunities to which you 
answered ''no" to the question on line 20, 
answer "yes" or "no" to the following question: 
"Is the child likely to answer incorrectly be- 
cause the material responded to is unclear, or 
because preyious material has not prepared 
the child to make such a response?" Sum the 
number of "yeses" to this question and enter 
this number. 21. 

22o Subtract the number of potentially inappropriately 
cued responses (20) and the number of potential 
error^ (21) from the total number of classroom 
response opportunities (19)« Enter this nimi- 
ber of appropriately determined response 
opportunities per classroom. 22. 

23. Divide the number of appropriately determined 
response opportunities per classroom (22) by 
the number of children with reading assign- 
ments iirifetrniSlassroom. Enter this number 

on line p3 and on the coding sheet. 23. 

24. Divide tne murnber of appropriately determined 
respons/e opportunities (22) by the total number 
of response-Qgportunities (19). Enter this 
percentage on liite 24 and on the coding sheet. 24. 
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Behavioral Analysis of Cur-ricula Data Sheet: Instructions 
Measures of Adapting Variables 

I. Consequence ratio . Diagnostic testing should save students' time 
by ensuring that they only study material that they do not already 
know. However, since diagnostic testing comes at a cost in stu- 
dent time, it is necessary to have a measure of the cost -effective- 
ness of such testing. The consequence ratio, which relates time 
spent taking diagnostic tests to total time (i. e. , diagnostic test 
time plus prescribed teaching time) is such a measure. To cal- 
culate each classroom's total diagnostic test "time," you will use 
information from. the in-class interview. 

Using information from in-class interview Items 68 and 69, locate 
each diagnostic test that a child in the classroom has taken. If 
the test is in a published curriculum, the interview data provide 
title and page numbers. If the test has been developed by the 
teacher, a copy of it with subject identification number is attached 
to the interview questionnaire. 

Next, examine each diagnostic test to select a unit to stand for test 
"times. " The word "time" is used because the most accurate con- 
sequence ratios are calculated by actually timing the students as 
they perform the diagnostic test and consequent material. However, 
almost as good an estimate can be made by selecting an artificial 
"unit" to stand for time. Since the result is a ratio, it is important 
that the unit counted in the diagnostic test material" be identical to the 
unit counted in the consequent teaching material. After examining 
the diagnostic tests and teaching material, you can select an 
arbitrary unit that seems appropriate for the curriculum. For 
example, the number of math problems the student does in the 
diagnostic test and in the consequent material might be counted * 
and compared. In a reading curriculum, the number of words in 
the diagnostic test might be compared to the number of words in 
the consequent material. As long as the unit counted in the diag- 
nostic test and in the consequent material is Identical, such 
things as number of. pages, lines, etc., may be compared. 

% 

Once the unit to be compared is selected, count the number of these 
units in each diagnostic test taken by each child in the classroom. 
You are interested in a classroom total of diagnostic test units or 
"time. " For example, given that diagnostic-* test A, containing five 
units, had been taken iDy two children, and that diagnostic test B, 
containing three units, had been taken by one child, the classroom 
total of diagnostic test units would be 13 (5 units x 2 children + 
3 units X 1 child). Enter this classroom number of diagnostic test 
units on the data sheet. 

187 



B-58 

In calculating the consequence ratio, it is irrelevant whether the 
student actually took or skipped the material. The ratio simply 
identifies the cost-effectiveness of diagnostic testing as compared 
to the potential time, spent in teaching and diagnosing. Therefore, 
each diagnostic test time included in the ratio mus.t have an accom- 
panying consequence time. Included in the consequence "time" of 
a diagnostic test are all readings, exercises, lessonjs^ tests, etc., 
that a student who fails the diagnostic test must take and that a stu- 
dent who passes the diagnostic test may skip. 

Usually in, individualized curriculum materials, the consequent 
'material for each diagnostic test is clearly identified in the teacher^s, 
manual. Thus, a consequence ratio [could be calculated by examin- 
ing the curriculum materials alone without reference to their use 
in a classroom. However, a teache^ frequently will use the 
materials in ways that vary from thej format identified in the 
teacher's manual. For example, if ^ teacher prescribes fewer 
assignments than the curriculum suggests consequent to perfdrmance 
on a diagnostic test, the actual classroom consequence ratio would 
differ from a ratio calculated by examining the materials alone. 

Classroom information alone cannot be used to calculate a con- 
sequence ratio, since there is no in- class record of the consequence 
for children who "skip" material. Thus, we suggest that you begin 
by identifying the developer's recommended consequence for each 
diagnostic test taken, and then correcting this figure with the avail- 
able classroom use information. 

For each published diagnostic test recorded on in-class interview 
Items 68 and 69, locate the publisher's recommended consequence 
material. Using the same arbitrary unit counted in the diagnostic 
test^ (line 1), estimate recommended consequence "time. " Remem- 
ber that we are concerned with the potential consequence of failing 
a diagnostic test. Thus, a consequence "time" must be entered 
for each diagnostic test taken by each child, even if some children 
in the class passed the diagnostic test and so avoided the consequence. 
For example, assume that diagnostic test A has 5 xmits with a 
potential consequence of 20 units. Mary passed and JoAnn failed 
test A, The total potential consequence time is 40_uiiits. 

For each teacher-developed diagnostic test (if any), locate material 
actually or potentially prescribed consequent to test performance. 
This material is also identified on the ;in-class interview. Items 68 
and 69. Count the units to estimate this consequence "time. '» 

Add the developer's recommended consequence "time" and the 
teacher-developed consequence "time" and enter this figure on 
line 2 of the data sheet. 
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To correct for teacher changes in developer recomxnendations, 
use data from the same in-class interview question. You can 
identify the actual prescriptions for children whose diagnostic 
test performance indicated the necessity for teaching material. 
Compare the actual prescription to the developer's recommended 
prescription in terms of number of units. If the prescriptions 
are identical for a child, note zero correction. If^, for example, 
a child's actual prescription is 5 units shorter than the recom- f 
mended prescription, note a correction of -5; if a child^s actual 
prescription is 7 units longer than the recommended prescription, 
note a correction of +7. Add all corrections, attending to sign 
in each case. The above three examples give a total correction 
figure of +2. Enter your total corr^sction figure and the appro- 
priate sign on line 3 of the data sheet. 

This correction figure is to take Qare of the circumstance in 
which a teacher gives diagnostic tests, yet assigns teaching 
material even to student^ whose test performance indicates a 
"skip. " Although, ordinarily, the consequence ratio is calculated 
without reference to actual student performance, in this unusual 
case the teacher *s modifications are actually at a cost in student 
time above a nonindividualized version of that curriculum in 
which the students simply take all the teaching material. 

To identify prescribing consequent to a "skip, " note each student's 
score on the diagnostic test (in-class interview Items 68 and 69) 
and use ^he developer^s recomr^endations to analyze that score as 
indicating a "skip" or "take. " If the teacher has prescribed 
material that students' scores indicate may be skipped, count 
the number of units in this prescribed material and enter this 
number on line 4 of the data sheet. 

1 

Correct the number of units entered on line 2 hy adding or sub- 
tracting (depending on sign) the correction figure entered on line 
3. Enter this corrected consequence number on line 5. 

Correct the number of units entered on line 5' by subtracting from 
it the number entered on line 4. Enter this a!s the fina^ ^corr ected 
consequence "time" on line 6 of tjie data she^t. / 

The consequence ratio is the consequence "time" divided by the 
sum of testing "time" and consequence "tim^. " To find this sum, 
add lines 1 and 6, 

Divide line .6 by line 7. Take this division to two significant 
places. Enter this number on line 8 of the data sheet and on the 
coding sheet. 
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"9. Pis cr iminability ratio, ' A diagnostic test should reflect individual 
differences. If all students pass or all students fail the diagnostic 
test, their prescriptions are identical. No individual differences 
have been detected and, thus, we might expect that no benefits of 
individualization would be detected. 

Using data from the in-class interview. Items 68 and 69, count the 
number of students who passed a diagnostic test, includirig both 
published and teacher-developed tests. Enter this number on 
line 9. 

10. Using data from the same in-class interview question, count the 
number of students who failed a diagnostic test, including both 
published and teacher-developed tests. Enter this number on 
line 10. ' 

11. The discr iminability ratio is the number of students who passed 
' or failed (whichever is smaller) divided by the number of students • 

in the class. Select the smaller number from lines 9 and 10. (If 
no students were given dia^gnostic tests, the "smaller" number is 
zero, and zero divided by the number of students in the class is 
still zero. ) Count the number of students in the class, including 
those who did not take a diagnostic test. Divide this. total number 
. of students in the class into the selected smaller number. Take ^ 
this division to two significant places and enter this number on 
line 11 of the data sheet. 

12. The discr iminability ratio as calculated in line 11 varies from 0 . 
to . 50. To make it more directly comparable to the consequence. 
ratio^and the predictive validity ratio, which vary from 0 to 1.0, 
it is simply doubled and entered on the coding sheet. 

13. Predictive validity ratio. A valid diagnostic test predicts whether 
students need teaching material before being able to pass the post- 
test or mastery test. The diagnostic test and mastery test purport 
to measure the same student behaviors. Therefore, the diagnostic 
test should be a good predictor of performance' on the mastery te^st 

^ if the student is n,ot given intervening teaching material. If no " 
mastery test corresponding to a diagnostic test can be identified 
in a particular curriculum, data derived from a second administra- 
tion of the same diagnostic test can be treated exactly as if from 
a mastery test. 

Attached to the in- class interview, Items 68 and 69, you will find data 
that have been gathered as follows, during a specified time period 
of approximately one month, the teacher was ask*ed to identify the 
first diagnostic test takejfii by each child and record the child^s per- 
formance on that.test. Then prior to any attempt to teach those 
objectives, the teacher administered, the corresponding mastery 
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test, or readmiiiistered the same diagnostic test. On the data 
sheet, location of the maste'ry test is identified and the child's 
performance recorded. 

Using a teacher's edition of the curriculum, check to see that 
each child^'s diagnostic test and mastery test do correspond as 
recommended? by the curriculum developers. In other words, 
this is not a judgment about whether the tests indeed measure 
the same' behavior, but rather a check on whether the teacher 
correctly identified the^ recoromended mastery test. Data from 
•children whose two te^sts do not "match" according to recommenda- 
tions of the developer cannot be used to obtain a measure of 
diagnostic test j^redictive validity. Eliminate all such "mismatches" 
from your data sheet. 

When a student^s diagnostic test performance indicates that - 
teaching material can be skipped, the prediction is that the stu- 
dent would pass the corresponding mastery test without any 
intervening teaching material. * Thus, for students whose diag- 
• nostic test performance Is a V^skip" and who also pass the mastery 
, test, the diagnostic test has correctly predicted mastery test per- 
formance. Count these students on the data sheet and enter this 
number on line 13. 

14. When a student's diagnostic test performance indicates that teach- 
ing should be taken, the prediction is that the student would fail * 
the mastery test unless he or she took some intervening teaching 
material. Thus,, for students whose diagnostic test performance 

is a "take'-' and who also fail the corresponding mastery test, the 
diagnostic test has predicted correctly. Count these students on 
the data sheet and enter this number online 14. 

15. Add the entries for lines 13 and 14 to find the nCimber of correct 
predictions. Enter this number. 

16. K tW diagnostic test indicates a "skip, " yet the student fails the 
mastery test, or if t^l^e diagnostic test indicates a "take," yet the 
student passes the mastery test, the diagnostic test has predicted 
incorrectly. AH decisions that are not correct predictions should 
fit in one of the above two ^categories. Using the data sheet, 
count the students for whom predictive decisions (skip-or take) 

were made. (Do not include students whose tests were mismatched > 
as explained in 13.) Enter this jiumber of decisions on line l6. 

17. The predictive validity ratio is the number of correct predictions 
divided by the number of decisionSv^(correct plus incorrect pre- 
dictions). Divide the entry for line 15 (correct prediction^) by 
the entry for -line 16 (total decisions). Take'this division to two 
significant places and enter this number on line 17 of the data 
sheet and on the coding sheet. 
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If yqu have a data sheet on pr'edictive validity for some but not 
all classrooms, you may enter the value for predictive validity 
derived from one classroom in the calculations for other class- 
roomspusing that same, cur riculum. 

18. Each measure oi the goodness of adapting can be correlated alone 
with end-of~year student achievement. However, direct relation- 
ships would not be expected since two programs could be equally 
high in dxscrxminability but differ greatly on the other two meas- 
ures. Yet, analyzing separately for each measure carrprovide 
information on how one factor contributes to overall achievement - 
if other factors are held constant. * It is also of. Interest, however, 
to get an overall estimate of the value of adapting. Since -the 
three measures are of interrelated factors, an improvement' in 
one measure comes at the cost of another. Tlje lowest value of 
the three, then, represents the effectiveness.^of that adaptive 
feature more accurately than does any kind of average. Enter 
the lowest value of the three ratios oji line 18 of the data sheet 
and on the coding sheet. ^ - 

Measure of Appropriately Determined Response Opportunities 

19. Information enabling you to locate each' child's assignment should 
be taken from t'he in-class interview. Items 68 and 69. If different 
children are working on different assignments, be sure to locate 
all of the appropriate materials. To.identify response opportuni- 
ties, you must use copies of the child^s version of the teaching 
material and, in addition, a teacher's edition. (A teacher's ^ 
edition alone is not sufficient even though the student materials 
are reproduced in it. These materials are reproduced with cor- , 
rect answers already marked, making it difficult to know what 
material the child actually uses.) 

O ne'e you hav-e^xreate^ child's version of each assignment and 
its corresponding directions in the teacher's edition, you may begin 
to identify^ individual response 6>portunities. A response oppor- 
tunity is an occasion for student behsivibr thkt may be directly 
evaluated- -reading aloud, underlining, refraining from underlin- 
ing, coloring, answering questions, etc. Occasions' for covert ^ 
behavior, such as silent reading, that cannot be directly evaluated 
are not included in a count of response opportunities. 

Counting response opportunities in materials designed for a mini - 
mum of teacher intervention. As a general rule, each answer 
representing a child's decision that could potentially be marked 
right or wrong is counted as a single response opportunity. For 
example, a typical workbook auditory perception ex^ercise presents 
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several pictures of obj-ects. The child circles the pictures with 
names beginning with a particular phonerne*. Each picture is 
counted as one response opportunity, since a{ each picture the 
child must decide whether or not to circle, and this decision 
can be directly evaluated. As another example, a typical work- 
book comprehension exercise presents a picture accompanied by 
'two words or phrases. The child circles the word or phrase 
that describes the picture. Again, at each picture the child 
makes a ^deciqion that can be directly evaluated as tight or wrong, 
so each picture^is counted .as one response opportunity. In another 
example, some workbook exercises may have a different number 
of response opportunities depending upon the specific directions 
for use. The workbook page may look like this: 




If the child is directed to circle each letter "e, the number of 
response opportunities is equal to the total number of letters .in 
the exercise, since the child makes a decision at each letter that 
can be evaluated. If, however, the child is directed to draw a 
line on the path that has onl^ letter "e*s" on it, there are fewer 
decisions to be evaluated' and correspondingly fewer response 
opportunities counted. In4:his case, the child either draws a line 
or does not draw a line on each -segment of the path between 
branches. .Thus, there are as many, response opportunities as 
they are between-branch segments of path, i. e. , sevfen. In 
another typical comprehension format used in later grades, chil- 
dren compose or select answers to questions based on their 
silent reading of- a story. In this, case, each question overtly 
answered counts as one response oppo^rtunity. 

Counting; f-esponse opportunities in materials requiring teacher 
presentation to a group of students . Much new material is pre- 
sented to students in this manner. All estimates of~response 
opportunities will be derived from analyses of directions in the- 
teachej;''s manuals, not from actual teacher performance. Teacher's 
editions of reading materials alv/ays make s6me suggestions on 
methods of presenting new material, and, in fact, some reading 
curricula come complete with detailed teacher scripts. Typically, 
in providing guidance for group instruction, the teacher's edition 
suggests that the teacher present some new information and then ask 
the children certain questions, or ''elicit" certair ^-esponses frorh 
them, or ask for a volunteer to perform the newly learned response 
or to answer questions about the just-read story. If the children 
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are to answer in unison, a response opportunity provided for 
each child in the group. However, if the tec^cher is directed to ask 
for a volunteer, or call on only one child, only one r esponse.^Gfp-^ 
portunity is counted. Since it is a total of all response opportunities 
that will be calculated-; arbitrarily assign this one response oppor- 
tunity to one child* s tally. 

Finding the clas sroom total of. response opportunities . When 
you have located each child's reading assignment and counted 
the* response opportunities in that assignment, simply add these 
per-child totals to find the classroom total of response oppor- 
tunities. Some value representing response opportunities must 
be entered for each child who has a reading assignment, even 
if that value is zero. Make a note of the nuinber of children 
involved for use in^item Z3. On line 19 of the data sheet, enter 
the classroom total of response- opportunities . 

In computing the value entered on this line of the data sheet, you 
must closely examine each response opportunity that has been 
entered in the total for line 19. You will make a judgment about 
the minimal behavior necessaryto achieve a correct answer for 
each response opportunity, * and a second judgment comparing that 
minimal behavior to the inferred or stated behavioral objective of 
the item. In order to make such judgments reliably, you must ^ 
have completed -the training keyed to this variable, appropriately 
determined response opportunities. The following instructions 
briefly outline the conc'ept o.f response contingency and review 
some guidelines for judgments. They arfe not K^substitute for 
the training of the curriculum experts. ' 

Response contingencies . A student learns what he or she performs. 
Usually only a small part of the student's activity is public ayid 
available to the instructor- -that is, a question is ans wb^-red about 
material the student has read. In well-designed materials,^ the 
student's successful public performance depends upon cp-rrect 
execution of the private act. The question can be answered <fbr- 
rectly only if the material has been read. In poorly designed 
materials, overcueing or. inappropriate cueing frequently allows 
a 'successful public performance based on trivial private behaviors, 
as when, for example, a question can be "answered" by/merely 
copying underlined word^from the text. In the workbool^ page 
illustrated earlier, the ^'e\Js"* are arranged in a pattern. If the 
directions are to circle each "e, " the responses are overcued, 
since the pattern as well as letter shape cues the response. If 
the directions are to draw a line through the path that has only 
"e's" on it, the response is not overcued. 
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* Group instructional eituatiohs are inherently flawed in terijis of 
^^^Ij^onse contingencies. One child in^the group may be respond- 
ing to the material and cues provided by the teacher, while 'otlier^ 
may simply remain silent or repeat the first child's response. 
Because of this contingency problem, ah apparently well-designed 
response opportT4nity in group instruction may not actually function 
as su^h. Thus, we generally recommend that group instruction 
response opportunities be "coimted" as zero. By this we do not^ 
imply that children do not learn in these group .situations, orfly 
that the learning that occurs cannot be simply and directly related 
to the response contingency aspect of the design of the curriculum. 

You may wish to make an exception to this general rule gjid count 
group instruction response opportianities in a particular curriculum 
if: (1) that curriculum provides detailed tfeacher instruction and 
training packages to ensure consistent classroom implernentation", 
andJZ) that curriculum provides the teacher with instructions about 
how to minimize the response contingency problem in group instruc 
" tioA. ^ - 

' Guidelines for evaluating privaf!e behaviors. Once you have identi- 
fied the behaviors tha^ will result in a correct answer for each , 
response opportunity, you must j'udge whether these are, ind'eed, 
the behaviprs that item was intended to teach. In many cases, 
.behavioral pbj^ectives for a lesson or worksheet -will be clearly 
and explicitly identified in the teacher's edition. In other cases, 
you must infer the objective by ex©,4mining the lesson or worksheet 
^.--Vas' well as some of the surrounding teaching material and the 
^""""^ general objectives usually stated at the beginning of the teacher's 
edition. ^ 

Once you have identified an explicit or inferred behavioral objec- 
tive for the response opportunity, and the range of behaviors that 
result in a correct response for that response opportunity, you 
may easily answer the question from line 20 of the data sheet, 
"Can a child respond correctly without behaving in the manner 
described by the stated or inferred objective?" Ask this question* 
of each response oj^portunity . Sum the number of "yeses" to 'this 
question and enter that number on line .20 of the data sheet. 

Just as it is important that a correct public* response should not 
be achieved by undesired private behavior^ it is also important 
to learning that the desired private behavior can and does occur. 
If response contingencies *are correct, student error rate provides 
a direct empirical measure of whether the desired private beha^vior 
has occurred. However, it is possible to estimate potential er- 
rors from an examination of the curriculum materials. The train- 
ing keyed to appropriately determined response opportunities 
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provides\gurdance in judging whether previous material has pre- 
pared the child to perform the correct response. You should 
complete this training before making the judgments calledffor 
in line 21 of the data sheet. If, however, it is possible for you 
to pbtain performance records of. the children in a particular 
classroom on the individual response opportunities you are examin- 
ing, you may substitute these empirical records for your judgments. 
(Brief guidelines on the- interpretation of error's in children's per- 
formances should be included in the training of curriculum ex5)erts.) 

However, it is most probable that you will s imply ^be making judg- 
ments about whether errors are likely on a particular item._ 
Remember that you are concerned only with those response oppor- 
txinities that are properly cued (those to which you answered ^'no^' 
in line 20). Answer '^es" to the question on line 21 if the item, is 
obviously fimbiguous (as are some of the pictures used in beginning 
reading exercises to 'teach audiforY discrimination), obviously - 
calls for complex behavior, that has not been practiced before '(as 
dp gome exercises in "just reading aloud" words containing untaught 
phoneme -grapheme correspondences), or a^sks the child to respond 
in an unsupported situation when earlier reJ^onses were improperly 
cued. Add the number of "yeses" to this question.and enter this 
number on line 21 of the data sheet. "t \ 

'T}ie. number of "'{yeses" entered online 20 represents the number *. 
of improperly cued response opportunities likely to result in cor- 
rect answers through undesired behavior. The number of "yeses" 
entered on line '21 represents the number of undetermined response 
opportunities likely to result in errors. Subtract each from the 
total number of response opportunities (line'l9) to find the class- 
room total of appropriately determined response opportunities. 
Enter this^number of line 22 olthe data sheet...; 

The numbef entered on tfne 22 ie the classroom total of appropriately 
determined response opportunities. To find the average number 
of appropriately determined respo'nse opportunities per child, you ^ 
mult divide this sum' by the number of childreai in the classroom, 
who had reading assignments". (Refer to your calculations for the 
number of^response opportunities [line 19] to find the number of 
children.) Enter the ave^a-ge number of appropriately ^determined 
response opportunities p^ child on the' data sheet and'coding sheet. 

In order to find the percentage of appropriately determined" response 
opportunities per class, divide the number of appropriately deter- 
mined t:esponse opportunities (line 22) by the totg.1 number of response 
opportunities (line 19). Enter this percentage on the data sheet and * 
coding sheet.. f 
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Behavioral Analysis of Cur riculaiCoding Sheet . 



1. 
2. 
3. 

4; 

5. 

• 6. 

7. 

8. 

9. 

10. 
11. 

12. 

13. 

14. 

15'. 

16. 

17. 

18. 

19. 

20. 

21. 

22. 

23. 

24. 



Leave blank 



Enter this number 
Leave blank 



Enter this number 
Leave blank 



Enter this number 



Leave blank 



/' 



/ 



Enter this number 
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Additional Information 

* 

Predictive Validity Ratio 

Some additional information regarding the predictive validity 
ratip is presented below. This' information can serve as the basis for ^ ^ 
a letter to br meeting with the teacher to explain this measure and the 
procedures that must be followed if it is to be estimated in his/her 
classroom. . As indicated previously, obtaining an estimate of the 
predictive validity ratio requires classroom intervention that may be 
difficult for the contractor to accomplish. Therefore, it should be 
regarded^as an optional measure. 

' What is predictive validity? Many curricula claim to a(Japt to 
individual differences in children through the administration of diag- 
nostic tests or pretests that are given before >any formal teaching of 
the objectives jneasured by those tests has taken place. a result 

of performance on the diagnostic test, a- child either skips or takes a 
'certain segment of teacher material. If the curriculum is to be truly 
adaptive, the decisions about what teaching material a ctdlcj is to re- 
ceive or slcip must be accurately made. 

There are potentially two types of diagnostic^ test inaccuracies. 
If through an invalid diagnostic test, a child is directed to take teaching 
material that he/she already knows, the child is wasting time that could 
be better spent dti other material. The- more detrimental error=^ccurs 
when a child, through an invalid diagnostic test, is directed to' skip teach- 
ing material that he/she really needs. The child may never be assigned 
to this mat6ri3.1 and, if it is critical to learning later material, may ul- 
timately flounder. , 

The predictive validity ratio provides a measure of the accuracy 
of such diagnostic test decisions. It is simply a ratio of the number of 
accurate diagnostic test decisions to total decisions. Surprisingly^ ap- 
plying this measure to a variety of curricula revealed tlmt many widely 
used diagnostic tests^iiaake invalid predictions. * 

Measuring predictive validity . In order to identify the accuracy 
of any single diagnostic test decision, a slight change in the normal class 
robm routine of prescribing, teaching, and testing is required. Ordi- 
narily, a child takes a diagnostic test, then skips or takes the relevant 
tesfching material and mdstery test in that order. The change in routine 
involves simply following the diagnostic test with the corresponding mas- 
tery test (or with a readministration of the diagnostic test) before giving 
any material designed to teach those objectives. After a child has taken 
-the diagnostic test and its corresponding mastery test (orMmgnostic re- 
test), the normal classroom routine resumes. The child'^^trffce s or skips 
the relevant teaching material according to the original performance on 
the diagnostic test. 
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This slight change in routine is necessary for only one diag- 
nostic test per child. During a selected four-week period, the first 
time each 'child takes a diagnostic test, the corresponding mastery 
test (o*r'diagn(j;s'tic retest)" can be iiranediately administered,, and the 
data recorded an the Predictive Validity Data ^heet on page B-70. 
At the end of this time period, the data sheet can be mailed*in or col- 
lected by the* interviewer. 

Please note that it is not, necessary tha^t each child in the class- 
room take a diagnostic test in the selected time period. Of course, 
the more children from whom data are derived, the more accurate the 
predictive. validity ratio will be, but at least ten children from any one 
classroom will be svifficient. Note also thaf the names of .individual ' 
children should not be included on the section of the data sheet re- 
turned to the interviewer. ^ 

Identifying the diagnostic and mastery tests . Once the time 
period for data collectionls selected, the children themselves will 
identify the diagnostic tests on which the data are>,to be collected. 
The first diagnostic" test taken by each child in that time period is 
^included in this study. , < 

Since children will probably be at different points in the cur- 
riculum in any one time period,' a number of different diagnostic tests' 
vdll be in the sample. Usually, diagnostic tests are clearly labeled 
as such in published curriculum materials. Each diagnostic test in the 
sample rnust be identified by noting its pag ft' numbers on the data sheet, 
'if the classroom uses teacher-developed diagnostic tests, enclosing a 
copy of each used in the sample' would be useful. Should thei^e be any 
question about whether &r not a particular test is diagn"ostic, the follow 
ing definition will be helpful: a diagnostic test is one taken prior tQ 
any attempt to teach the objectives measured on it arid one that has 
diffe^rential consequence depending on studeijt performance (i.e., 
students either take or skip a particular segment of teaching material). 

Mastery tests are often also clearly labeled in published indi- ~ 
vi dualized materials. When the published cul-riculum labels and pairs 
a diagnostic test with a corresponding mastery test, it becomes simply 
a matter of ensuring that the appropriately labeled mastery test is 
administered to each child who has taken the related diagnostic test.^ 

In some curricula, however, mastery tests are not so clearly 
paired with diagnostic tests. It is sometimes pos'sible to identify a 
section of a larger posttest that correlates directly with the objectives 
tested by the particular diagnostic test. This section of the larger^ 
posttest may then be administered after the diagnostic test. 
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Predictive Validity Data Sheet 
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If there is any doubt about "'whether such a segment really cor-- 
relates with the diagnostic test, it>is,better not to use it in this study. 
When an appropriate mastery test cannot be paired with a diagnostic 
test, the same diagnostic test shoald be readministered and the child's 
retest performstnce recorded on the data sheet under the "Mastery 
Test'* column. * ^ 

If the classroom uses tekcher-developed mastery tests, en- 
closing a copy of each used in this sample would be useful. If the 
mastery tests used.were published in the curriculiim, their page nirni- 
bers should be noted on the data 'sheet. (If the procedure was, "diag- 
nostic test and retest, " the same page number ''should be recorded in 
both places on the data sheet. ) - ^ 

Time between the two tests . Some time should be allowed to 
elapse between the two test administrations, especially in the "diag- 
nostic test and retest" procedure. We suggest allowing at least one- 
half day between the two tests. Beyond that constraint, the amount o£ 
elapsed time can be a matter of teacher convenience. However, it is 
vital that a child not be exposed to teaching* material directed at the 
tested objectives in the time period between the two tests. However, 
in this interval, each' child may any other teaching material not 
"directed at objectives covered by that child's diagnostic test. 

Information That Can Be Obtained from the In-Class Interview 

Note: Student name is not necessary for the curriculum expert. 
We prefer an arrangment that preserves student anonymity, yet does 
not mix individual student data. 

Info rniation for Questions 1 and 9 

Last diagnostic test taken,. Published test or teacher - 
developed? Title and page n\imbers if from published 
text. Copy of test if developed by teacher. 

^> 

Score on this test. 

Did student skip material consequent to performance 
on this diagnostic test? 

Was student prescribed material consequent to per- 
formance on this test? Title and page n\imbers pre- 
scribed if from published text. Copy of exercises 
prescribed if. developed by teacher. 



1. 

2. 
3. 

4. 
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5. Fill this out only for those students who skipped 

material based on the teacher -developed diagnostic, 
test- -What material would have been prescribed 
consequent to a ''take'* score on this test? Identify 
page numbers in published text and include copies 
of teacher -developed^ exercises. 

6. ' Place a checkmark by those student^^ for whom no 
(j diagnosis by means of formal testing has been 
attempted. 

Information for Questions ?3 and 24 

1. Each child^s last in-class assignment (not test). 

a. . Page numbers of assignment in the pub- 

lished text. 

b. Copies of assignment, if constructed by- 
teacher. . 

c. If some- children had a teacher -led group 
lesson, page numbers in the teacher's 
edition of the published text. 

d. If tape-and-workbook combination, identify 
tape so that a transcript can be located by 
the curriculum expert. 



202 • 



t 

APPENDIX C 
VIDEOTAPING 



, - Page 

# 

Directions for Producing Videotapes * . C-1 

Directions for Obtaining Measures from 

Videotapes C-11* 

Directions for Coding Videotape Information 

(including Videotape Summary Sheet) C-34 
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Directions for .Producing Videotapes 



I. Equipment 

There are many\systems available that may be used to videotape 
a classroom, but only one set of recommendations is made iti 
this Appendixi Any comparable pi:oducts may be used. 

Four basic components are necessary. The following list gives 
the name of the 4evice and a recommended model and /or size. 

Video Recording Unit. (Sony, 1/2 inch reel, 
Model 3600) 

Video pamera and Tripod. (Sony, AVC 3200) 

Microphone Systefn. (Sony, wireless. A micro- 
phone, receiver, and earphone are necessary.) 

Video Monitor. (Sony, 9 inch) 

In addition, four tapes of. one hour each are required for each, 
classroom. This would mean that for 400 classrooms, 1,600 rolls 
. of one-hour videotape are needed; Scotch Color Compatible 1/2 
inch tape is recommended. 

A portable collapsible cart on which to arrange and carry the 
equipment to the classroom, and on which the audio equipment 
can be set up for the taping sessions, is also valuable. 

II, Sending and retrieving tapes from sites * 

Tapes are sent out to the sites twice a year, in the fall and spring. 
The -Field Coordinator should have each tape labeled on the video- 
tape casing as well as on the tape reel. Pertinent information, 
to be filled in by the persons making the tapes in the classrooms, 
should include the site, teacher's name, subject area, and date 
of taping. 'The labels might look like the following illustration: 

SAMPLE LABEL 

Site: 

Teaser: 

Subj: . _ V 

Date; J 

It is necessary to send two one-hour tapes for each classroom in ^ 
the fall and two more in the spring. Accompanying the tapes sent to 
a site should be a letter to the Site Coordinator cjnd to the teacherg., 
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explaining briefly the purpose, of the videotaping sessions and the 
basic procedure that will be used, and 'emphasi?:ing the confi- 
dentiality of the information to be gathered (see page C-3). One 
or more return-address labels to facilitate the return ctf the tapes 

after completion should also be included. 

♦ 

The Field Coordinator must keep a list- of the dates on which the 
tapes are sent out and the dates/on which they are r^fe«4^ed in 
order to make sure that all tapes are retrieved. A letter ]^3ta*y be 
sent to Site Coordinators and teachers to show appreciation far^heir 
cooperation during the videotaping (see page C-4). 

III. Videotaping personnel , , 

A team of two people is needed for actual taping. At least.one 
person must be well trained in the use of the videotape equipment 
* (the taper), and the other person should have at least a minimal; 
understanding of how the equipment is used (the assistant). The 
taper is fesponsible for setting up the equiprnent in the cjlassroom, 
choosing the location from which .to tape, and doing the can;iera 
work. The assistant is responsible for scheduling taping sessions 
to match the schedule and convenience of the teacher; explaining , 
the procedures to the teacher; assisting the tapep in setting up the 
equipment; monitoring the audio and yideo portions of the tape 
through the use of the video monitor and earphones; keeping time 
using a stopwatch (to tell the taper when to start and stop a scan); 
and acting a.s a liaison between the taper and the teachers in order 
to make the teachers feel as comfortable as possible, 
• ^ ■ ' r 

There a^e two possible ways of at,ructuring the "team. " First, 

the team may be f?rom the hoi:ne-base of the contractor and sent " . 
out to the various sites from there. If this is the case, more^ 
than one team will be necessary. Assuming that two classrooms 
can be tapjed in one day by one team (this may be an over estiijlnate) , 
and it is .ajdvantageous to complete all 1:aping in two weeks, then 
20 teams would be needed for 400 classrooms. The s.econd alterna- 
tive is to use teams from the sites. The Site ^Coordinator or school 
personnel could possibly be used as members of the videotaping 
team. . 
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Sample Letter to Site^to Accompany Tapes 

r 



Return Address * 

Date , * • 

Dear Site Coordinator and teachers: 

As you know, your classrooms have been selected as part of 
a study of individualized instruction that is being sponsored by the 
National Institute of Education. In accordance with the plan for data 
collection, we will be sending our trained personnel to yoXir school 
to videotape the classrooms selected. 

Videotaping procedures are used to evaluate classroom proc- 
esses and have been found to be a mobile and reliable data collection 
device. The information that will be obtained will not be used to 
evaluate any individual teacher, student, school, or district, and all 
data will be considered to be strictly confidential. 

Enclosed in this package are the Videotapes that will be used 
by our taping team. Each classroom will be taped for one hour of ' 
mathematics instruction and one hour of reading instruction* Please 
store the tapes in cCsafet dry place to insure that they are not damaged. 

The taping team will contact the Site Coordinator shortly to 
arrange a schedule that will be' convenient for all concerned. 

' We would like to thank you ahead of time for your support and 
cooperation in this effort^ and to agaifi assure you that anonymity 
and confidentiality will be maintained. 

Sincerely, 



Sample Letter to Site after Completion of Taping 



Return Address 

Date * • 

Dear > ; 

We have now completed videotaping in your school. 
This ef/^rt was part of an evaluation of individualized programs. 
In order to assure anonymity, ^ the results of the evaluation will 
be reported by, grade level, not by Individual classrooms or by 
schools. Thank you for your patience and cooperation in this 
impo^rtant effort. We especially appreciate the courtesy, that 
you extended to us while we were in your school. 

Sincerely, 
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IV. Specific directions for videotaping in classrooms 

The aim of this portion of the Appendix is to give specific direc- 
tions to the team that will be videotaping in the classrooms. 

A. School and classroom behavior of the videotaping team 

■When entering a school, it is important that any personnel 
look and act in a professional manner. 'Following a few 
simple guidelines will aid in acceptance and facilitate co- 
operation. This does not imply relinquishing individuality 
while in the schools, but it does mean that courteods and , 
considerate behavior is expected. These guidelines are 
suggested: 

1) Dres¥ appropriately. Jeans and T-shirts are not accept- 
able in most schools. If possible, check with* the Site 
Coordinator to determine the mores of each particular 
school, as they may differ greatly. 

2) Be considerate of school parking facilities. Try to un- 
load all equipment at least a half-hour before the stu- 
dents arrive. This will allow ihe use of convenient door- 
ways. After unloa.ding, be careful not to park in some- 
one's assigned space, or to block other cars or entrances. 
If the school lot is small, park on the street. 

3) When arriving, report to the Site Coordinator and/or office 
in the school building. Each school has different procedures, 
but most require that external personnel sign in and sign 
out. 

4) -Set up and organize all materials early. Taping during • 

the first class period requires setting up in that room 
before students arrive. 

5) If the teacher^s room is use d for waiting, leave it in the 
same way it was found. If there is a coffee maker, leave 
money (usually 10^ a' cup) for any coffee that you drink. 

6) Never talk^ about a child, teacher, class, or any other 
school personnel while in the school. Remember that as 
an outsider, personal views will not be appreciated. 

7) Never discuss another school while on site. This can 
only lead to discomfort of those at the pres'ent site, as 
they will be concerned about what might be said about 
them at the next site. 
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8) Remember that the school is a self-contained world with 
its own values, problerhs, joys, and conflicts. What is 
said and done in one world with little fear of misrepresenta- 
tion cannot always be said or done with similar consequences 
in the school'? world.* • 

' 9) If any difficulty arises, therii^ should be a back-up system 
to cover it. For example, there should be a duplicate set 
of eqmpment rea^ in case of an equipment failure. Last- 
minute scheduling difficulties should be handled by the Site 
Coordinator or principal if the coordinator is unavailable. 

10) Be c|.9 efficient as possible. Set up and disassemble equip- 
ment* quickly and quietly. Try to cause as little disruption • 
to the school processes as possible. 

11) Maintain a professional approach at all times. Be friendly 
but not "chummy.^' Respect the teacher's position as the 
authority in his /her classroom. Respect the principal's 
authority as head of the school. And most of all, respect ^ 
the enormous load that all school personnel must handle. 
Try to be flexible and calm no matter what happens. 

12) Maintain distance with the children. Being nice does not 
rriean being patronizing or motherly. Do not enter into 
contact relations with the children (e.g., eye contact 
with "meaningful smiles, " patting), unless it is neces- 
sary for the performance of a task. 

* . ' 

13) Be open and accepting of any. suggestions made by school 
personnel. If the suggestions do not conflict with the in- 
terest in obtaining precise data, try to be as accojcnmo- 
dating as possible. 

14) If teachers or other school personnel ask questions about 
the videotaping, be polite, answer briefly, -and try not to 
discuss the matter in any great detail. 

15) Be appreciative. Thank all involved for their time and 
assistjs^Tice. 

16) Remember to sign out or to let someone know ypu arfe^^ 
leaving when you depart froln the school building. 

17) Always drive slowly and carefully near any school build- 
ing. 
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Videotaping procedures 

1) ^Setting up / 

As previously stq/ted, the videotape equipment should' be set 
up in the classrooin as early as possible before the actual 
taping session. If possible, the camera and crew should be 
in the room for one hour prior to taping so that the teacher 
and students can become accustomed to the equipment. If 
two classrooms are to hp taped in one day, e. g. , one clg^ss- 
room in t^e morning and one in the afternoon, the equipment 
can be seli up in the first room before the students arrive in 
the morning and in the second jroom wihile the students are at 
lunch. However, scheduling will not always be so simple, an( 
the taping team must be flexible and able to rnove the equip- 
ment quickly and quietly. ( 

All equipment should be arranged in the classroom in a posi-^ 
tion where it dofes not inconvenience the teacner (ft children ' 
or block an entrance, but still permits visual and auditory 
'\ accuracy. Care should be taken in the placement of extensio 
cord wires. 

* Avoid placing the camera so tljat it is aimed directly ^t 
windows or lights, as this can distort the video portion of 
the tape aryi even damage the vidicon tube in tlie camera. 
If no other position is available, request that the blinds be 
pulled so that the camera is not facing into direct s\mlight. 

The camera should be mounted on a tripod to facilitate 
smooth movement. Once the equipment is sefup, plugged 
in, and turned on, check and adjust the camera focus and 
F-stop, and Be sure that the. camera position permits an 
/ tinobstructed view pf the entire classroom. 

Audio reception cair readily be checked by having one team ^ 
member (probably the taper, as the assistant will be moni- 
toring iudio signals) travel the^fuU perimeter of the room 
with the microphone while the other team member checks the 
range and adjusts the receiver for maximum clarity (using 
the earphones). 

2) Scheduled time ^ 

J 

Each teacher should be taped teac}iing-one hour of reading 
and one hour of mathematics. If the class rojom is under 
open or mixed scheduling (so that reading and n;iath"emaElTrs . 
ar.e taught at the same time), two cohtinuous hours should 
/ be taped. If a subject is. taught in periods lasting less 
than a continuous hour but there is more than one period 
of that subject per day, tape both periods. 

., 2.10 
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3) Micropjione 

Immediately preceding the videotaping session, ask 
the teacher to put 'on the microphone. A wireless micro- 
phone is preferred, as it is more convenient and unob- 
trusive, but wired microphones could also be used. In 
either case, the microphone should be worn in lavalier 
^ fashion, usually about eight inches below the teacher's 
mouth. Request that the teacher remove any jewelry 
that might clank against the microphone as the teacher 
^ /moves; y^ns or necklaces create noises that distort 
the a^dio portion of the tape. . . 

The teacher's statements are the primary concern of the 
audiq/^Jortion of the tape, and the students* verbal inter- 
actions with the te^acher are only slightly less important. 
It is mandatory that a clear', codable record of those 
verbal behaviors be made, with a minimum of other 
auditory interruptions. 

4) Camera 



When the teacher, students,, and videotaping team are 
,ready, the taping session should begin with the assistant 
' / starting the stopwatch. The taper should focus on the 

teacher and any student or students with whom he/she 
is interacting for the first five minutes of the tape. The 
, assistant will tell the cam'era person when that time is 
up. The camera then pans across the room for two 
minutes, focusing on each child for approximately fiv^- 
/ seconds. The purpose of this scan is to determine 

whether each child is actively engaged in cognitively '/ 
related material. It is not necessary (or even desirSLble) 
for the person with the stopwatch to indicate each five- 
second segment; that can be estimated by th^ camera 
person. However, at the end of the two jninutes, the 
assistant will indicate that termination. It is important 
to emphasize at this point that all communication between 
the two yideotape team members must be nondisruptive 
and preferably nonverbal. 

Following those seven specified minutes, the camera 
should resume following the teacher's movements and 
should include in the frame the chil^or children with 
whom the teacher is currently interacting, if possible. 
After every five minutes, another two minute scan of 
the entire class should be made, focusing again on each : . 
child for about five seconds. Five minutes of following 
the teacher is interspersed with two minutes of all the 
individual students throughout the entire taping session. 
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5) General instructions . ^ ^ ' i 

a) Th^ tapeheada on the video recorder should be 
cleaned thoroughly ptior to eaph day of t^ping^, 
and between sessions if nec,essar^. 

b) When moving the capaera to follow' the teacher or 
scan the classroom (called "panning**), be sure to 
move the cameifa slowly and evenly; much data are 
lost through popr camera work, so take time but 
stay with the subject. 

c) If the camera has a zpom lens, change from close-up 
to full-scene shots infrequently and srnoothly. Try 
to keep the image roughly the same size. for most! 

of the taping. ^ 

d) It is usefuX to Ihe person who will observe the 
tapes if the persoh operating the caineca scans 
the entire room for a brief^eriod (30 seconds) 
at the beginning of the tapii^ session, so that a 
general impression of the room amangement and 
size can be obtained. 

e) BE CERTAIN THAT THE NAME OF THJE TEACHER 
IS ON THE VIDEOTAPE REEL, AND, IF POSSIBLE, 
ON THE AUDIO OR VISUAL PART OF THE TAPE AS 
WELL. This can be done either by making a placai>d ' 
containing the inforrnation and focusing on it at the 
beginning of the tape long enough so that it can be 
read, or by saying the information into the micro- 
phone before giving it to th^ teacher- (be ^sure the 
recorder is on! ). 

6) Checking the tapes 

After each taping session'is complete, play back several 
portions of the tape to be sure that both; the audio and 
video portions are recorded clearly. The focus and con- 
trast should be clear, the teacher's and students' voices 
should be audible with a minimum of static or distortion, 
there should be no^^'snow'' or other visual ^distortion, and 
the identifying information for the teacher," site, subject, " 
and date should be on the tape casing, reel, and, if possible,' 
on the tape itself. If the tape is unusable, schedule re- 
taping as soon as possible. 

- • / 
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Mechanics of quality control 

As) soon as all tapes are completed for one site,, they are to be sent 
back to the Field Coordinator using the box in which they were 
sent and the return address label provided. As soon as they are 
received, the Field Coordinator should check each tape from be- 
ginning to end. The purpose of this preview is to assure codable 
technical quality. In other words, the tapes must be clear both 
in sound and image. 

Once previewed, acceptable tapes should be checked off on the 
master list and the date they were received should be noted. 
If some ta^es are unacceptable, the Field Coordinator should note 
the reason(s) why they are unacceptable. The poor quality tapes 
should then be erased and sent back to the site with a cover letter 
explaining the difficulty and requesting a new taping s'ession as ' 
soon as possible,. The date that any such tapes are sent back to 
the sites should also be noted on the master list. 
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Directions for Obtaining Measures from Videotapes 

i 

I. Collecting information from videotapes 

This portion of the Appendix is of primary concern to the videotape 
oljservers and those who will train them, and includes five parts: 
< time unit, observation unit, guidelines for categorization, directioiis 
' for observing students on task,, and guidelines for ratings. To observe 
a videotape, the video recorder and video monitor should be set up in 
a stable position at eye level in a quiet room. A clipboard with. coding 
sheets can be held on one's lap or put on the side-arm-of a chair. In 
this position, if one is right-handed, the^stopwatch can be held in the 
left hand and a pencil in the right. The stopwatch is simply a time- 
keeper ^nd should not be stopped unless the tape is also stopped. < . 

A. , Time unit 

The Videotape. Observation Sheet (see page C-12) is divided into 
five blocks, each block representing a one minute time period. 
The one minute block is then further divided (dotted lines) into 
fifteen second time segments for each observation unit. The 
teacher's vei:bal behavior as well a^s the student's response will 
be watched for a fifteen second time segment. In the five seconds 
immediately following the fifteen second segmeijt, the actual re- 
V cording takes place. Aft^r watching the tape for fifteen seconds, 
the observer marks the appropriate categories either during 
the fifteen seconds (this becomes easier with experience) or 
during the five second break. 

B. Observation unit 

There are three observation units that are used'to denote with 
whom the teach^er is interacting. The first symbol, "I," is for 
' ^ an individual student. "Gs*' refers to a^small group, which can 

be any size from two to one child less than the whole class. 
(This does not include children who leave the classroom for 
any reason, who are set aside from the body of the class for 
discipline, or absentees.) "G all" means that the teacher is 
talking to the whole class. The observer must decide whether 
the majority of each fifteen second block of time is directed 
^ towards an individual, a small group, or the whole class, 

C. ^ Guidelines^ for categorization 

Each observation sheet has twelve categories across the top of 
the sheet. - The observer must be completely familiar with the 
definitioi^s and examples of each catefgory prior to^ observing, 
since he/she ijaust make continuous, reliable decisions. The 
following is a list of symbols used and their meanings. 
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Videotape Observation Sheet 



C-12 



Coder 
Teacher . 
School 
Grade 
Subject . 
Date 



fo Whom 
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1) - = negative 

A negative statement may be either management* briented, 
/ personal, or, cognitive. It may be stated in the form of a 
question, exclamation, imperative, or declarative. Any 
statement is considered -negative if the quality of what . 
is said is essentially punitive. Sometimes it is not the 
words that convey the punishing nature, but the tonal 
quality used by the teacher/ "You did very well on that 
work,'" if said in a sarcastic and degrading manner, may 
^ rnean that<j3ie teacher is telling the child how terrible 
he/she is doing. ^ - ' 

In a sense, negatives^are the opposite of praise^nd are 
seen as negative motivators. Many negative statements 
involve the demand to stc^p a be^vior; for example, , 
"Sat down, " "Stop it, " and "You weren't listeriing, " Re- 
^fferring to repetition is another common use of a negative: 
"How many times do I have to tell you what to dp?" Others 
may imply a negative aspect of the child's overall per- 
formance: "You never pay attention, " arid "Dan't you ey^r 
read the directions first?" ^ * 

The negative column is checked whenever a negative 
occurs, although only one check is permitted foB a fifteen 
second block* The observation sheet on page C-14 shows 
* some samples of possible negative patterns. The first 
minute shows that the teacher made a negative manage- 
ment statement to an individual child. He/she did the 
same thing during the second minute even though he/she 
was working with a sm^l group in cognitive material for 
the majority of the time. During the third minute,* the 
teacher was working with an individual child or children 
but made a negative management to the whole class during 
the first fifteen seconds. The fourth minute indicates 
that the negative was cognitively related. 

2) M - management \ 

Managen^ent is any. statement" ar question made by the teacher 
to an individual child of children that contains no cognitive 
information but that serves a managerial function only. 
It may concern discipline or personal functions. Some 
general examples are: "Yes, you may go to the bathroom," 
"Please sit down," and I'Get your pencil sharpened." 
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Videotape Observation Sheet 
(Examples of Negative Behavior) 



Coder . 
Teacher . 
School . 
Grade 
Subject . 
Date 



To Whom 
Directed 
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3) CM ~ cognitive management 

Cogljitive management is any statement or question made by 
the teacher that is managerial but cognitively related. It 
may deal with curricular materials or preparation for a task; 
for example, "Turn to page 10, ^' "Has your work been checked 
yet," and "Get your counting sticks out for that problem. " 
If "good" or "right," etc., oc^rs by itself as the 
teacher looks at a child's work, it is considered a cogni- 
tive management statement. If "good" or "right" occurs 
in conjunction with specific feedback, it j.s then considered 
a cognitive^ statement (category 5); for example, "Good, 
you did these division problems^ correctly. " 

4) O = other ^ 

In the preceding categories, a check was marked if any 
such behavior occurred during a fifteen second block of 
time. ■ In this category, a .check is marked only if the 
• majority of the fifteen seconds was spent in neither 
cognitive nor management interactions. This includes 
personal comments and tangential information not specific 
to the materials. Examples of this type of verbal behavior 
are: "How is your brother," "You have a dog, too, don't 
you, " and "Do you ever lielp around the house like John 
is doing in this story?'* * 

5) CS = co'gnitive statement 

This type of verbal behavior should be marked if it occurs 
at aijy time within the fifteen second time block. A CS occurs 
when the teacher makes a cognitive statement that doers 
not elicit a response from a child or cl;^ildren. It may 
be a statement that requires merely an echo response 
from the child, asking a rhetorical question, lecturing, 
giving a co'mmand, or giving cognitive feedback; for 
example, "2 + 2 = 4, what is 2 + 2," "Read the sentences, 
then fill i*n the blanks using your new vocabulary words, " 
and "If 4 jumps of 3 spaces each =#12, then 4 times 3 = 
12. " It is important to emphasize here that even a ques- 
tion regarding cognitive material is considered a cogni- 
tive statement Ji the child is not expected to respond, 
or in fact does not respond. 
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6) CQ = c cognitive question . ^ 

Mark this symbol on the oTDservation sheet if , during 
* the fifteen second blogk, the teacher was inter- 
acting >?V!ith a student or students when a response was 
elicited.i This includes asking ^ direct cognitive question 
such as VHow much is 2 + 2," or making a statement sut:h 
as "Rea4 me the sentence that tells what Jane is doing, " 
It may also include asking the child. to read the directions 
of what he/she is supposed to do. 

Another t^pe of cognitive question may at^'first appear to 
be managerial (M), but cognitive information is clearly 
being passed on; for instance, "How many pages did 
you do todays," "How many were you supposed to do," 
"How many are you going to do today, " and "Did you do 
more or less than you had pla'nned to do?" Questions 
that give detailed discussion about the titles of books, 
how to spell them, and how to write them out^ although 
they, are teaching children skill's not normally taught in 
the curriculum, ^re considered cognitive questions or 
statements. 

7) RES = response 

This category is checked on the observation sheet if a 
child or group of children either initiated a cognitive 
question or statement, or they responded to pne. If 
it was child initiated, circle the check; if not, just check 
the appropriate block. Responses may occur in the "1," 
"Gs, " or "G all" rows, depending on who is responding.' 
"Gs" and "G all" responses are usually choral. 'In order" 
for a response to te child initiated, it must be clear and 
obvious such as a child going up to the teacher to/ask - 
a question. Raising one's hand is not necessarily enough 
of a signal. 

8) SIL = silence 

If, during the majority of the fifteen seconds, the teacher 
or children were not ^aps^-king, it is counted as a*silence. 
This may occur while the teacher is checking work-, the 
children are doing seatwork, etc. 

S 

9) TIM = time ' , f> 

If the tea^fher interacts with an individual f or 60 seconds 
in cognitively related material, or if the teacher has three 
consecutive* checks in the *'CS" column with either a small 
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group (Gs)'or the whole class (G all), then tlie time 
category is checked in the fifteen second block at the 
end of the 60 seconds. In other-words, if the teacher 
is working with a group, a ^^time^' is checked if he/she 
is lecturing^ which is counted on the observation 
sheet after three consecutive "CS*' checks. If, on 
the other hand, the teachej^ is working with an indi- 
vidual student, the checks may occur in the '^CS" or 

the "CQ" column, and the interaction is a tutorial one. 

f 

ERR =^.error 

This box is marked on the observation sheet when the 
teachej makes an error that goes uncorrected. The 
error may be factual, in + 2-= 6,** or it may be 
an extremely noticeable pedagogical error (that is^, an 
error that reflets a very poor leaching technique). 
An example of this type of error is ^'What word sounds 
like 'Mike*," where the expected response is "five.** In 
this case, both words have the long "i** sound but the, 
words themselves do not sound alike, and the child may 
be more confused by the error than he/she was before 
the question. " , - , . . 

CHA = change 

A check mark ( /) is made in the change column of the • 
observation sheet when the single student or group of 
students^ at the beginning of a fiftefen second block is 
different from the one{s)' with whom the teacher was in- 
teracting at the beginning of the previous time block. 
Changes are only coiinted when the interaction relates 
to cognitive material. The observation sheet^Mi4)age 
C-18 shows an example of a teacher who was working 
with Johnny during the first block of time and for half 
of the second block, H|^/she moved to Judy during the 
second block and worked with her during the third. The 
^change was noted during the third block of time , since ^ 
at the beginning of the second fifteen seconds he/she 
was still with Johnny. : 

If the teacher is working with a small group, the number 
of students in that group is noted in the space to the right 
of the change column under "N6tes.'' On the sample 
observation sheet on page C-18, the teacher is working 
with a small group containing four students during the 
second minute. 
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Videotape Observation ^hcet 
(ExamplM of Chengtos) 



Coder . 
Teacher . 
School . 
Grade . . 
Subject . 
Date 



To Whom 
Directed 




M 


CM 
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cs 


CQ 


RES ' 


SIL 


TIM 


ERR 


CHA 
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G all 
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During the third minute, the teacher be^n working with 
another small group, and the change was checked in the « 
first fifteen second block as well as the numt^^B of students 
in that group, which was noted under "Notes. 

12) Notes 

The "Notes" column is used to record the number of children 
in a small group as well as anything else that may be 
relevant to a more complete understanding of the tape 
for future analysis of the data. The observer may use 
this section to quote an ambiguous comment or possible 
error so that he/she gan go back and look at it again 
or ask for assistance in determining the proper notation 
^ to be used. * 

D. ^Directions, for observing students on task 

During the taping session, the camera will focus for two 

minutes on the classroom rather than on the teacher, scanning 

each individual child for about five seconds. The scan of 

child by child will occur every five minutes through the 

entire tape. The normal observing of the teacher's verbal 

behavior will continue durTng this time, even though the 

camera is not on him/her. However, a special notation ^ 

will be made of the children during the two minute scan to 

determine the percentage of children on tas k. It should 

be remembered that the purpose of this measure is to estimate 

the opportunity that children in a classroom have to learn 

the information^esented. The definition is, therefore, 

very restricted! It does not mean that the activities that 

are recorded as off task are not valuable and ifriportant. 

Definitions 



On Task! child actively engaged in work with materials 
that directly relate to the acquisition of subject-matter- 
related information. 

Cleai:ly On Task; child sitting at d^desk or table or on 
the floor apparently reading, writing, or counting objects. 

Clearly Off Task: child playing v/ith blocks, sand, toys, 
or another child. Child moving from one place to another. 
Child sitting with hand raised or otherwise "waiting. " 
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Ambiguous; if a child is talking with another <;hild, try to see 
if the two children arre clearly working on an academically related 
task. If so, consider it on task; if not, consider it off task. 
If a child is ^^getting ready'^ to work by opening up a book or 
arranging a task, consider it off task unlessi the child begins to 
wqfk and stops arranging. If the teacher is »lecturing'V and the 
child is obviously listening or trying to follow some set of direc- 
tions, consider it^on task. If the child is looking around or 
staring off into sjpace, consider it off task. >Any child who is 
trying to respond to a direct or group directed question by the 
teacher is considered on task (even if thife includes hand waving 
and shouting out answers). 

The n9tation for each two-minute section should be as follows: 

First Pass Second Pass ■ Third Pass Fourth Pass 



X 


X 


o 


- X 


o 


X 


o 


X 


X 


X 


X 


o 


X 


• X 


X 


X 


X 


X 


o 


X 


o 


X 


o' 




X 


• 


X 


• 


X 




d 




o 




X 




X 


f 


X 




X 




X 





An "x'» means a child was on task. An "o" means off/ If two or 
more children are together at once, make a decision for each 
child, in the group. 
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E. Guidelines for ratings 

At the completion of observing a videotape, the observer must 
fill out a rating scale (see page C-22), There are twelve 
rating categories with a scale of 1 to 5. It is essential that 
the observer be completely familiar with all of the categories 
and their definitions prior to watching a tape. Although the 
rating scale form is not filled out until the observer is finished 
^ viewing the tape, he/she may take notes in order to remember^ 

the pertinent details for rating each teacher. It is also dmportant 
to know the categories well so that when an event occurs, such 
as modeling a response, the observer will be aware that it is 
a piece »of information he/she will need to use later. Most of 
the ratings deal with the frequency of the occurrence of a behavior, 
and, therefore, the scale is from "1" (never) to 'V5" (frequently). 
Two of the ratings, however, do not require a frequency esti- 
mate; for management and tutoring, a quality judgment is 
required. The following is a list of the twelve categories and 
their definitions: 

1) Active responses sought 

The teacher elicits active responses from the students. 
That is, stuc^ents are required to verbalize, manipulate, 
write, or perform some other active form of behavior when 
the teacher. interacts with them as opposed passive be- 
havior such as listening. 

2) Teacher models responses 

The teacher very clearly shows the child what a response 
consists of. For example, the teacher says, ^'3+4 = 7. 
Write the seven in the box. " He/she then shows the child 
how to do it, and stands and watches as the child does 
the next problem. . . 

3) Teacher refers to earlier curricular information 

The teacher^mentions, during an interaction with a stu- 
dent, information that was previously learned; for 
example, "You sounded out words like this before/' 
It is sdrde communication to the child that the task 
the he/she is facing is something not totally new. . 

4) Teacher focuses child's attention on task 

This may be judged not only by what the teacher says or 
, dpes, but also on what the child is doing* That is, does 

the child look off into space or at the materials? Does 
• • the teacher let the student wander off on a tangent? Does 
he/she get the student back on task? 
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Videotape Ratings 



Teacher 
School 



Subject 



Frequently Never 

1. Active Responses 5 4 3 2 1 

• .Sought - ' ^ 

2. ' Teacher Models ^5 '4 3,21 

Responses 

3, Teacher RefeTrs to 5 4 3 ' 2 ' 1 
Earlier Curricular^ 

Information 

4, Teacher Focuses 5 4 3 2 1 
ir Child's Attention 

on Task 

4 5. Teacher Solicits 5 4 3 2 1 

Child's Opinion on 
Correctness of 

Responses - . ' ' 

6. Teacher Refers to 5 4 3 2 1 * 
Earlier Success 

7. Teacher Uses 5 4 3.2 1 
» Contingent Praise . - it^ 

8. Teacher Uses , 5 4 3 2 1 
General Praise 

9. Tutoring 5 4 3 2 1 

10. Management 5 4.3 2 1 

11. Feedback 5 4 3 2 1 



12. Teacher Interacts 5 4 3 2 1 

with Individual 
Students 
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Teacher solicits child's opinion^on correctness of responses 

The^teacher asks the child to determine whether his/her 
response is correct. 

Teacher refers to earlier success ' 

The teacher mentions something the child has done well in 
the past; for example, "You really did well with multiplica- 
tion yesterday; division is the reverse process. See if you 
can figure out how to do these problems. You have done 
this kind of thing very well in the past. " 

Teacher uses contingent praise 

The teacher praises a student for work that is well done; 
for example, *'You added these^tWo numbers correctly.- 
Good. Or, it may simply be telling the child that he/she 
consistently did something correctly. It is contrasted 
with the next category, No. 8. 

Teacher uses general praise 

The teacher praises a student in a nonspecific way: 
"You are working well, " "You have done good work, " 
and "You are in a great frame of mind today. " .The 
child must interpret what is being praised. No. 7 
is specific in relation to the behavior and No- 8 is 
general. 

Tutoring 

This judgment is based on the clarity, conciseness, and 
accuracy of the teacher*s tutori9.1s. That is, is he/she 
explaining things clearly, understandably, and logically? 
Is he/she to the point? Daes he/she answer the child 
quickly ancj flexibly? Does the qhilci seem to follow the 
explanation and use the materials appropriately? If the 
student doesn't understand, does the teacher take a new 
approach, become more concrete, or find some innovative 
way of expressing the thought? ^ 

Management 

This judgment is based on how Y/ell the classroom works, 
not necessarily on how low the noise level is. This cate- 
gory was developed because it was clear that despite the 
fact that teachers havev^y different management pro- 
cedures, some being quite constrained and strict and 
others being quite loose, within each type there are 
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very different successes. That is, there are plassropms 
where the teachers are very structured and want a quiet 
room but are not always telling the students to be quiet. 
Equally, there are classrooms that are open, free, and 
, quite noisy and in which the teacher seems to be quite 
comfortable. What is basically being measured is the 
degree of harmony between what the teacher seems to 
want and what the children seem to be doing. Is the 
teacher fighting whatever is going on in the classroom? 
Do the children seem to know what is expected of them? 
Do they get materials easily? Is everyone functioning 
individually as well as in a unit? Is the process smooth 
and even without major disruption? 

11) Feedback . ' ^ 

This judgment is based on the clarity and frequency with 
which the teacher gives corrective information to a stu- 
dent about that student's academic behavior. The informa- 
tion should be given in such a manner as to encourage the 
student to use the information to alter his /her behavior . 
In other words, the teacher gives the student information • 
with which the student is able to make correct responses 
in the future. This type of teacher behavior usually occurs 
in tutorial situations with an individual student durjng 
which the teacher questions, the student responds,- and 
the teacher gives feedback on that response, 

12) Teacher interacts with individual students 

This judgment is based on the degree to which the teacher 
interacts with individuals within the class as opposed to 
dealing with small groups or the entire class. In other 
words, ^ in a strictly individualized classroom where the 
teacher only interacts with individual students, the rating 
would be "5. " In a strictly traditional mode^. where/ the 
teacher deals with the class as a whole, the rating would 
be 'a, " 

Training videotape observers 

To assure clear perception and reliability among observers, the 
training of those oTDservers should be done in a highly systematic 
way. Only one trainer is necessary, but that person must be fully 
familiar with the observation schema prior to beginning training 
sessions for others. The trainer should also have built up his/her 
own intrarater reliability by that time to assure clarity in the 
training and ability to answer questions about items that could 
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initially appear to be ambiguous. Both the trainer (who may also 
function lat^r'as one of the observers) and the observers can be 
trained by the same methods, and all will be referred to. as 
observers in the following description. 

Each step in the training procedures that follow is ^xplainod 

in general with specific suggestions for activities. Each session 

is expected to last approximately one hour. 

A. First session: Introduction 

The first session should be an introduction to the observa- 
tion of videotapes. This session should' include at least 
. the following: 

1) An explanation of how and why the videotapes were 
collected, 

2) A brief description of what the observers will be looking 
for on the tapes, and how the training process will pro- 
ceed. 

3) Copies of the instructions for observing tapes, to be 
given to each trainee and read by him/her • 

4) A brief explanation oi each category. Try to avoid 
"What if . . . questions in the first session. If there 
are questions regarding the substance of a category, 

' answer them to the best of your ability, but do not 

attempt to cover all possibilities in the first session or 
confusion is likely t^ result. 

B. Second session: Equipment and time 

For the second session and a^ll- following, a full set of equip- 
ment for viewing videotapes will be necessary. This includes: 
a video recorder, video monitor, electrical connecting cords 
(interequipmentj, take-up reel, and a videotape of a class- 
room. Each--©b^erver should have his/her own stopwatch and 
clip board. - 

1) ExpldiTvand demonstrate how the machinery is set up and 
connected. Have the observers examine)the machine and 
each trainee set it up. / 

Demonstrate- the threading of t^e tape onto the take-ixp 
reel, and operate the control switch to start, stop, and 
rewind the tape. Each trainee should also attempt this 
until he/she has mastered the procedure. 
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3) Give each observer a stopwatch and demoiistrate how ^ 
to start, stop, reset, and wind it. 

4) Give each* trainee a blank piece of paper and a pencil. 
All observers should now watch the tape for timing. 
Everyone should start his/her stopwatch when the 
trainer says "start." Tell the observers to make 

a check on the paper after every fifteen second block 
when you say "mark. " They then have five seconds 
between each block of fifteen seconds in which to mark 
their paper. Continue this way for three minutes. Then 
have the observers mark their papers independently 
for the next three minutes (do not announce when to 
"mark"). 

At the end of the three minutes, have everyone stop 
again and count their marks. Each observer should 
have nine checks on his /her paper. This exercise is 
intended to familiarize the observers with the process 
^ of using and estimating the fifteen second intervals, 
and marking the papers (later, observation sheets) 
within the five second interval. 

C. Third session: Management and cognitive statements 

I^he third session is meant primarily to distinguish between 
teacher comments that are managerial and those that ar% 
cognitive. In this session, you will not be concerned with 
the subdivisions of each group (that is, M vs. CM, CS vs. 
CQ, etc. ), nor should you try to distinguish to whom the . 
teacher directs his/her corrunents (that is, Gs, or G all). 
The equipment for this session includes the videotaping equip- 
ment- from the list session, and a tape of a classroom. Each 
observer should have a paper with two columns drawn on it, 
one column marked "Management" and one marked "Cognitive.'' 

1) Discuss the difference between a teacher's managerial 
statement and a cognitive statement. Emphasize that 
majiagement statements usually deal with a student's 
functioning (e. g. , "Has your work been checked, " 
"Get a pencil, " and "Please sit down. "). Cognitive 
statements usually deal with subject matter (^. g. , 
"What is 2 + 2," and "Read the sentences carefully. "). 
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2) Begin to play the tape as in the previous session, telling 
the observers when to start (for synchronization purposes). 
Continuing for three minutes, tell the observers how to 
mark the cplumns (tell them whether what occurred in the 
time segments was managerial or cognitive), for each fifteen 

S second time block. If both a managerial and a cognitive- 

statement occur in a fifteen second time block, indicate 
that both columns are to be checked? for the interval.- 

3) Watch the tape for another three minute time span, but 
this time the observers should mark the papers at the ^ 
appropriate times individually, without prompting from ■ 
the trainfer. 

4) Discuss the notation of the last three minute time segment. 
Correct any errors and clarify ambiguities. 

5) Have the observers watch and record their observations for 
another three minute segment by themselves, recording 
management and cognitive statements on the observa- 
tion sheet. 

D. Fourth session: Differentiating management and cognitive 
management 

Follow the same procedures as in the third session (that is, 
discuss the definitions of the tdrms, give examples, make the 
distinctions clear, have the entire group observe together, 
have the observer trainees observe independently, check the 
observation sheej;s togethei; and discuss errors and ambiguities, 
and repeat a three minute session pf independent observing). 
The videotape may be replayed at any time to clarify what was 
said by the teacher or'to show an example of a particular verbal 
behavior. 

E. Fifth session: Differentiating cognitive statements from ^ 
cognitive questions 

Repeat the same pr ocedur e *f oUoWed in the third and fourth 
sessions (C and D), this time emphasizing cognitive s. 

F. Sixth session: Combine all managements and cognitives ^ 

By this session, all the observers should understand the 
cognitive and management categories thoroughly. If not, 
proceed only with those observers who have demonstrated 
their knowledge of those categories by recording them accur- 
* ately. For those individuals still having trouble differentiat- 
ing the two kinds of verbal behayiofs, separate sessions 
should be held to deal with their difficulties. 
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For those who are prepared for this sixth session, prepare 
papers on which there are four columns of categories to record: 
M, CM, CS, CQ. Follow the identical training procedure, 

• using a videotape as. in the previous sessions, but increase 
the observing times to five minutes. After the trainees 
have completed the second independent observing block, 
check the rate of agreement in each of the four categories. 
This is done by each observer summing the number of checks 

* in each of the categories and comparing them with the trainer's 
sums. For example: 

Trainer Observer 

M CM CS CQ M CM CS CQ 

3 5-1,9 2'"* 6 1 10 



Percent Agreement; 

M CM CS ^Q 

' 2 5 j. '9 

3.6 1 10 

67% 83% 100% 90% average = 85% 

Any percentages that are below 80 are areas of concern. The 
average percent agreement will indicate which observers are ^ 
having the most difficulty, and may require additional training. 

G. Seventh session: Negatives 

Proceeding in the exact format as session F, now add a 
"Negative" column to the observation sheets (ybu will now have 
five columns reading -, M, CM, CS, CQ). It is possible that 
no negative statements will be encountered in the tape, but 
an awareness of negative verbal behavior should be emphasized. 

H. ^Eighth session: Othejr and silence 

Gradually Increasing the range of^categories that the observers 
must keep in mind while viewing videotapes, proceed as in 
previous sessions, with emphasis on the new categories ("Other 
and "Silence"). Keep. in mind that these categories are marked 
only if they take up the rriajority of a fifteen second time block. 
Extra practice may be hecessary to master this distinction. 
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Ninth session: Responses 

Proceed as in previous sessions, but increase coding times 
to seven minutes. ' 

Due to the format in which taping is done, each observing 
sequence will now probably run into or through one of the. two 
minute scans of the entire classroom (these occur throughout 
the tape at five minute intervals). Explain clearly to the 
trainees that at this time, their primary concern is with the 
teacher's verbal behavior, .not with the change of camera focus. 

Clearly explain thelftwo types of response behaviors: Simple 
response's are checked (/), but child -initiated responses are' 
checked and the chdck is circled ( (/))• 

Tenth session: Time and error 

As with negatives, an error may not occur during this 
session. When either an error or a negative does occur, 
however, it should be noted and replayed so that all observers 
have a thorough understanding of the occurrence. Continue 
this session in the previoijsly cited format. 

Eleventh session: Change and notes ' 

Add these last two categories and proceed as before. . 
Twelfth session: To whom directed 

B^gin using the pbservation form (see page C-12) and proceed 
with the training session as before. . The observers will now 
have to decide to whom the teacher's verbal behavior is directed 
for the majority of the fifteen second' time unit. 

Thirteenth session: Practice I • 

With this session begin to try to develop reliability between 
the observers and trainer. This is strictly a practice session. 
Begin by filling out the identifying information requested^in the* 
top right hand corner of the Videotape Observation Sheet. All 
observers should start their stopwatches at the same time, and 
then observe at least fifteen minutes of tape independently 
(everyone can be together in oi>e room). Break for a discussion 
of any problems and then observe iot another fifteen minutes. 
Where the trainer -trainee agreement is severely off, private 
sessions may be necessary to correct the observer's errors. 
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N* Fourteenth session: Practice 11 ^ 

This session is basically a repeat of Practice L However, 
when observing is finished, introduce the Videotape Summary 
Sheet (see page C~35) and have the observers fill out the 
first 24 items for the 30* minutes' total time they have just 
observed. 

O. Fifteenth session: Practice III 

r Repeat th'e Practice II session. Check the percent agree- 
ment in' each category using the sunamary sheet information 
'of each observer and comparing it with the trainer's summary 
sheet. Each observer should calculate his/her own average 
percent agreement. 

At this point, some observers may be finished with their 
^ training while others may need more time to practice observ- 
■ ing or to backtrack to a category with which they are having 
difficulty. Proceed to the next .step as the observers reach 
an acceptable reliability level. 
^ ' ■ 

P. Sixteenth session: On task 

The purpose of this session is to familiarize observers with 
the observing'of the two minute classroom scans throughout 
» • the videotapes. This recording on observation sheets should 
be done simultaneously with the recording of the t,eacher's 
•verbal behavior. Find a place where such ^^-two minute scan 
begins just prior to beginning the training i^ession. 

1) Explain and discuss the rules for recording. *^on task*^ 
and ''off task" behavior. 

2) Start the tape ^nd tell the observers when to make an 
"x" and when* to make an "o" (on task and off task). 

3) Advance to the next two-minute- s can on the tape. Have 
the trainees record by themselves, marking an "x" or 
an "o" for each child. 

4) Discuss any errors on the trainees' observation sheets 
and clear up any ambiguities. Replay and re-record 
any t\^b minute scan if necessary. 

♦ ■ 

Q. Seventeenth session: On task and verbal * 

1) Review "on task" recording and have the trainees 

record their observations on another (new) two minute 
scan. 
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2) Go to the next two minute segment and have the observers 
attempt the simultaneous recording of the "on task^' infor- 
mation and the teacher's verbal behavior. 



and make explanations where necessary. 

4) Advance to the next two minute scan and repeat the sim\il~ 
taneous recording. 



R. Eighteenth session: Practice IV 

All the observer-trainees should observe an entire tape now 
(approximately one hour) for both te9,cher*s verbal behavior 
and students* "on task" behavior* ^ Check all observers* reli- 
ability using the procedure on page C-Z8. 

At the end of these practice sessions, some observers may 
still need more practice than others to achieve acceptable 
reliability, and they should be given opportunities to practice 
on their own as well as with the trainer. ^ 

S. Nineteenth session: Ratings • ^ » 

This session should familiarize the observers with the twelve 
ratings to be completed after watching a videotape (see page 
C-2Z for the rating sheet). Discuss the definitions of each 
category, explain the rating scale, then watch a videotape 
for about thirty minutes to extract information for the rating 
sheet. Rate the classroom together and clarify any problems.'^ 

T. Twentieth session: Final practice session 

Observe an entire videotape and record all information on the 
observation and rating sheets. Check interobservation agree- 
ment. 

Reliability on the ratings is determined by summing the absolute 
difference between the observer's rating and the trainer's rating 
for each of the twelve categories, and then dividing the sums 
• by twelve to get the average absolute difference. An acceptable 
average is anything less than 0, 50. 
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Check the observation sheets for errors or ambiguities 



Example : 



Absolute 
Difference 



Trainer 



Coder 




Tutoring 
Management 
Feedback 
etc. 



4 
3 
2 



3 
4 
2 



1 
1 
0 



.67 
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Some observers may now be ready to begin observing tapes 
on their own for actual usable data. Other observers may 
need to have one or more practice sessions to clear up any 
problems they might have. For those who are vinable to re- 
cord teacher verbal behavior and '^on task" behavior. at the 
same tirtie, an alternative is to have them observe the tape 
twice, each time for different information. The ratings can 
be done after both observation sessions are finished. • Thia; 
is more time consuming than simultaneous observing, but 
improves accuracy. 

Reliability 

A. Reliability of observers 

Several aspects of the data collection are designed to insure 
interobserver reliability. The basic training of observers is 
- conducted using a program specifically designed to teach 
reliable videotape observing. The training program includes 
step-by- step procedures with frequent checks that insure a 
thorough understanding of all categories. Built-in checks on 
the reliability of observing between the trainer and the observer 
guarantee uniformity of interpretations. Further, continuous 
checks are recommended during observing of actual tapes for 
usable (not training- session) data. This is best implemented 
by having the trainer or a different observer watch again every 
tenth tape completed and record the information from it. Inter- 
observer reliability should be carefully calculated,' and any 
significant disagreements may then be reviewed and observed 
together by both persons at a second viewing of the tape. 

Checking intraobservei* reliability involves having each ob- 
server watch again a tape he/she has already watched and 
recprded. This should be done routinely after every twentieth 
hour of observing. If tapes are recorded differently by the 
same person, a second person should watch and record the 
tape to resolve differences. 

B. Teacher stability or generalizability 

The stability of classroom behaviors over time can vary for 
different categories. Psychometric theory assumes no changes 
in the behavior being measured. However, teachers* behavior 
can change between the fall tape and thie spring tape. Some 
behaviors are more stable than othCTS. (For example, the 
difference between teacher cognitive statements and manage- 
ment statements is more stable than a teacher^s question- 
ing style. ) OtJ^er behaviors change systematically at different 
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times of the year. (For example, the need to give detailed 
management instructions deci*eases over the year.) 

Variables that tend to fluctuate a lot have been eliminated 
from the observation schema, but those that change sys- 
tematically over time have been retained. By averaging 
the systematically changing variables, the discrepancies 
of measurement can be minimized. Only those variables 
for which the fluctuations of stability can be effectively mini- 
mized or readily explained are included in the observation 
schema for this study. 
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Directions for Coding Videotape Informatipn 



Observatio^n summary / 

The Videotape Siimmary Sheet must be filled jout for each tape. This 
form (see page C-35) provides space for all identifying' information 
as well as how long the tape v/as and a summary for each cate^ry. 
One form is used for mathematics and another for reading. 

After the first fall tape has been observed, fill in all the identifying 
information. How many minutes of tape were observed can be most 
easily, determined by counting the number of minutes filled in on^the 
observation sheets. If an entire hour of tape was observed, there 
should be twelve observation sheets (each sheet provides* space for 
five minutfes of tape). 

If less than one hour of tape was observed, the following' sums must be 
adjusted so that they are equivalent to one hour of observation. For 
example, if 50 minutes of tape is observed and the sum for a given 
category* is 25,:the proportion used would be: ^ 



This converts-to x 



_50 
60 
25' 60 
50 • 



25 

X 

The number derived from this formula 



'Would be the one recorded on the sumjnary sheet. 

For Negatives, Management, Cognitive Management^ and Time , sum 
those categories that were checked in {he individual row. Then sum 
the small group and whole class rows together and put the totals in the 
correct categories on the summary sheet in the ''fair* column. 

Oth^r, Silence, and Error are a straight total across group size unit. 
In other words, it doesn't matter if any of these three categories oc- 
curred when the teacher was interacting with an individual, a small 
group, or the Whole class. 

Three of the categories. Cognitive Statement Alone, Cognitive Question, 
and Response , mu»st be summed within each size -group. Therefore, 
there will be one sum for Cognitive Statement Alone /Individual, another 
for Cognitive Statement- Alone /Smgil Group, and so on. Cognitive State- 
ments Alone, however, will only be counted when they occur in a fifteen 
second block that does not contain a Cognitive Question. In the example 
below the Cognitive Statement Alone is counted for 'B, but not for A. 



CS CQ 



RES 
/ 



B 



CS CQ 



RES 
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, Videotape Summary Sheet 
Fall Tape--# Minutes Coded Teacher 



Spring Tape--# Minutes Coded 



Negatives 

1. Individual 

2. Group' 
Management 

3. ) Individual f 

4. Group 
Cognitive Management 

5. Individual 
6» Group 

Other 

7, Total 

Cognitive Statement Alone 

8, Individual 

9, Small Group 

10. Whole Group . 
Cognitive Que stion 

11. Individual . 

12. Small Group 

13. 'Whole Group 
Response 

14. Individual j 

15. Small Group 

16. Whole Group ^ 
Child "Inii;;ated Responses 

17. Total 



School 
Grade 



Subject 



Fall 



Spring 



Total 
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Fall ' Spring ' Total 

Silence ^ C . 

18/ Total ^ _i ^ J 

Time 

19. Individu9Ll lJL_ 

20. Group ^ . ' 

Error 4 

21. Total " ^ _^ 

Cognitive Contacts 

22. Individual \ 

23. Small Group ^ 



24. Ratio of Cbgnitive to 
Management Statements 

25. Percent of Students on Task 

26. Active Responses Sought 

27. Teacher Models Responses 

28. Teacher Refers to Earlier 
Curricular Information 

29. Teacher Focuses Child's 
Attention on Task 

30. Teacher Solicits Child^s 
Opinion on Correctness 

31. Teacher Refers to 
Earlier Success 

32. Teacher Uses Contingent 
Praise 

33. Teacher Uses General 
Praise 

34. Tutoring 

3 5 . Manag eme nt ^ 

3b. Feedback 

37. Teacher Interacts with 
Individual Students 
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Child- Initiated Responses are those cognitive responses that are 
circled on the observation sheet. They ^will always be individual, 
so a simple count of all circled responses is all that is required 
on this item. 

Cognitive Contacts /individual is the sum of the individual change 
checks plus 1 (in order to take into account the first child can- 
tacted)/ Cognitive Contacts /Small Group is the sum of the number 
of children in small groups, which is recorded in the *'Ndtes" 
column on tlie observation sheet. Therefore, for the sample 
shown in the second section, page C-18, Cognitive Contacts/Indi- 
vidual = 3, and Cognitive Contacts /Small Group = 7, for the four 
minutes of tape coded in the example. 

The Ratio of Cognitive to Management Statements is derived by 
adding the number of all Cognitive Statements Alone (including 
individual, small group, and whole group) to the number of all 
Cognitive Questions, and dividing by the sum of all Management 
and Cognitive Management statements. 



After the two passes, fall and spring, simply total the two columns 
in the '*Totar^ column for items 1-23. For the ''Total*' column of 
Ratio of Cognitive to Management Statements (item 24), agajji 
calcTilate the ratio from the total information on CS, CQ, M, and 
CM using the formula mentioned above. 

Scan summary- 
Using the information retrieved from each two minute scan through- 
out the tape should result in approximately eight passes per hour of 
videotape. Each tape is summarized to determine the percentage 
of students on task. To determine this percentage, sum the 
*'x's'* of all eight passes and divide by the total number of observa- 
tions* » • ( 



item 25. To obtain the percentage for the total colurmi, average 
the fall mathematics percentage with the spring mathematics ^ 
percentage. Do the same for the reading figuresT 



CS + CQ T M + CM 
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Rating summary 

The ratings within "each subject area must also be recorded on 
summary sheet, items 26-37, and averaged over the fall and 
spring passes. Simply sum the two ratings in each category 
and divide by two, rounding to one decimal place. For exampl 

Fall Mathematics ' ^' 
Active Responses Sought 5 

Spring Mathematics 
Active Responses Sought 5 

In this case, the recorded total rating for "Active Responses 
Sought" in mathematics would be 3. 5. , 

At the end of the year, there should be two summary sheets for 
each classroom: one for mathematics and one for reading. 
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APPENDIX D /\ 
TEST REVIEWS 



■ > 

^ Page 



Comprehensive Tests of Basic Skills . . ; D-1 

California Achievement Test D-6 

Metropolitan Achievement Test , D-9 

SRA Assessment Survey D-18 

Stanford Achievement Test. " D-24 

/ ^ 

Animal Crackers D-27 
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GENERAL INFORMATION 

Title: Comprehensive Tests of Basic Skills 
Author: None reported 

Publisher: CTB/McGraw Hill, Monterey, CA 
Publication date: 1974 

Forms: The CTBS achievement batteries consist* of seven overlapping 
levels appropriate for use with the following grade levels: 



Level 


A 


Grades 


K.O - 


1.3 


Level 


B - 


Grades 


K,6 - 


1.9 


Level 


C 


Grades 


1.6 - 


2.9 


Level 


1 


Grades 


2.5 ^ 


4.9 


Level 


2 


Grades 


4.5 - 


6.9 


Level 


3 


Grades 


6.5 - 


8.9 


Level 


4 


Grades 


8.5 - 


12.9 



There are alternate forms, S and T, for levels 1-4. Forms S and T 
are parallel forms in content and format, and have been statistical-ly 
equated. Levels A, B, and C are available only in Fortn S. 

The skills areas and tests on each level appropriate for use are 
as follows: 



Level A 

Alphabet Skills 

1. Letter names 

2r Letter forms 
Visual & Auditory Discrimination 

1. Visual discrimination 

2. Sound matching 

3. Letter sounds 

4. Listening for information 

5. Language 
Mathematics 



Level C 

Reading ' 

1. Reading Vocabulary 

2* Reading comprehension: sentences 

3. Reading comprehension: passages 

Language 

1. Language expression 

2. Spelling • 

3. Language mechanics 
Mathematics 

1. Mathematics Computation 

2. Math Concepts & Applications 
Science 

Social Studies O A 1^ 



Level B 

Reading 

1. Letter sounds 

2. Word recognition I 

3. Reading comprehension 

4. Word recognition II 
Language 

1, Language I 

2, Language II 
Mathematics 

1. Matheihatics Concepts & Applications 

2. Mathematics computation 

Level 1 

Reading 

1. Reading Vocabulary 

2. Reading Comprehension 
Language 

1. Spelling 

2. Language mechanics 

3. Language expression 
Mathematics 

1. Mathematics Computation 

2. Mathematics Concepts 

3. Mathematics Application 
Reference Skills 

Science 

Social Studies 
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Level A is a pre-reading test and assumes no school experience. 
Level B assumes approximately one year af instruction. 
Level C assumes approximately two years of instruction. 

Manuals and other technical aids: 

(1) Examiner's manuals (Provided for each level, 60-100 pages each) 
(2J ' Test Coordinator's Handbook (one for all levels, 93 pages) - 
(3) Technical Bulletin (73 pages) 



PRACTICAL CONSIDERATIONS 



Costs for Levels A and B 



Complete Battery (Reading, Language, & Kath) MS 
Complete Battery (Reading, Language, & Math) HS 
Scoring/student 

Costs for Level C 

Partial battery (Reading, Language, & Math) MS 
Partial battery (Reading, Language, & Math) HS 
S cor ing / s tuden t 

Costs for Level 1 

Partial battery (Reading, Language, Math and 

Reference Skills) MS 

Partial battery (Reading, Language, Math and 

Reference Skills) HS 

Scoring/student 



$16.45 pkg/35 
9.80 pkg/35 
.60 



$16.45 pkg/35 
9.80 pkg/35 
.60 



$16.45 pkg/35 

10.50 pkg/35 
.60 



Administration and Timing 

The publishers of the CTBS have taken great care in the preparation 
of the examiner's manuals appropriate for each level. The manuals are 
well written, provide explicit directions, and ^contain all the necessary 
information for teachers to administer and interpret the CTBS. 



All tests are timed. The suggested testing schedule for administering , - 
the complete and partial batteries of interest to our study are as follovrs: 

■» 

Levels A (144 min.), B (157 min.), & C (170 min.) - 4 mornings 
Level 1 (173 min) -3 mornings • 



FORMAT AND LAYOUT 

The format for all levels of the CTBS is of exceptionally high quality. 
The illustrations are clear and accurate, the print very readable, and 
the layout carefully avoids crowding of items and the confusion of 
inconsistent movement through rows and columns. 
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ITEMS AND THEIR COVERAGE . , ^ , 

Items for Levels 1-4 of .the 1974 Edition were derived from Fonns Q and R 
of the CTBS (1968-1969) and additional items reflecting current trends 
.in curriculum prepared by ^content specialists and CTB/McGraw Hill staff. 
Two new subtests. Science and Social Studies, were added to the batteries 
of Levels C through 4. For^ incliision in the battery, an item*was subject 
to the following criteria: i 

1. Difficulty index in the range from .25 to .90 (mean difficulty 
• for all items is .63). Difficulty indices for all items are 

reported. • ^ * 

2. Consistent ability to discriminate between high and low scoring 
students s 

3. Point biserial correlation (r^^) greater than .20. 

4. No bias against blaqk and Spanish-speaking students as judged 
by reviewers from these respective minority groups. 

The development of Levels A-C began by having teachers critique existing 
primary level achievement tests. Items for the primary edition (Levels 
A-C) were then written by teachers and curriculum specialists in the 
respective content ^eas. Item tryouts were then conducted on a 
standard sample of students and, in addition, on a special sample of 
3,000 students from schools having a minimum of 90 percent black 
St udents.>r- Black and Spanish-speaking educators reviewed the items 
3nd deleted tho'se it^s wftich discriminated against these respective 
minorities. \ 



CONSISTENCY 

The Technical Bulletin presents the data derived from studies of the 
internal consistency of each of the Levels A-4. Kuder-Richardsori 
Formula 20 correlations are respectable (i.e., >.75)*for almost all 
subtests but are, as would be expected, higher at the higher levels 
(1-4). ' ; 

Pearson 'product-momfent correlation coefficients! were computed as a 
measure of the extent to which successive level's of the CTBS measure 
the same thing. Adjacent levels (e.g.. Levels A and B, C and 1) 
. were administered to the same groups of studenljis at two-week to' 
five-month intervals. The results* of this study of interlevel 
articulation of pertinence to our purposes are as follows:^ 

Range of Interlevel Correlations for Total Battery 
Ad j acent . Levels (5-month interval between testings) ' 

A & B .59 to .78 (.78 for total) 

B & C ; .45 to .76 C. 76 f 01^ total) \ OAr 

C & 1 .52 to .80 (.80 for total) , 



This indicates that adjacent levels of the CTBS do, to a considerable 
extent, consistently measure the same thing. 



VALIDITY 



Content validity of the CTBS for our purposes can be determined T^y 
matching the curriculum objectives of individualized programs in our 
study sample with the process and content objectives published for 
each of the subtests of the CTBS. These process and content objec- 
tives' for the CTBS dre set forth in detail in the Test. Coordinator' s 
Handbook.' 

The intercorrelation coefficients between total battery scores on 
CTBS/S and total IQ scores derived Ifrom Short Form Tes^ of Academic 
Aptitude (SFTAA) are reported as folla.js: 

CTBS 



SFTAA Level 1 Level 2 Level 3 ' Level 4 

Level 1 .64 , 

Level 2 .79 .82 A 

Level 3 . . .85 ^ .79 

Level 4 • .82 .77 

Level 5 .79^ 



SCORES AND NORMS • . ' . 

^ ■ 

The national standardization and norming of the CTBS was .based on a 
probability sample of 130,000 students in grades K-12 drawn from 
public and Catholic school's in the 50 states. The narm sample was 
stratified along the following dimensions: 

kf.J School Type 

1. Greater Cities Public (N=37,650) 

2. Other Public (N=84,567) . ' ' 

3. Catholic\(N=8,742) 

B*. Geographic Region (U.S. Office of Education regions) 

C. Average Enrollment Per Grade 

1. Small (N<385) N-46,108 

2. Medium 0N<1,923 "+ N.2:385) N=34,482 \ 
I 3. L^rge (N^l,923) N-50,369 * . 

D. Community Type (Urban, ' Town, Rural, Other) 

The ethnic composition which resulted from this sampling procedure is 
as follows: 
• . ' 1. Black - 16.7% 

2. Spanish speaking - 7.9% 

3. Nonminority -"74.6% 

This represents a slight- overrepresentation of minorities as compared 
to the ethnic composition of "public schools as reported by the U.S. 
Office for Civil Rights in 197Q. 

« 

Separate norms^ f or large cities available upon request. 
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ADMINISTRATION AND SCORING 

Administration of the CTBS is clearly and explicitly detailed vin the 
Examiner^ s Manual . No other special training is required. 

Test batteries may be hand scored or machine scored. Answer sheets 
available for use with the CTBS include CompuScan, Digitek, IBM 1230, 
and Scoreze. 



COMMENTS 

The CTBS is highly recommended by this reviewer as being, in many 
ways, a model achievement battery. The tests themselves are superior 
as far as format, layout, validity, and reliability are concerned. 
The development of the CTBS has been carefully documented, and all 
claims *for the battery are supported by factual data reported in the 
^ Technical Manual. 

The following is a suggested schedule for using the CTBS in the 
proposed study. 



Grade 1 
Grade 2 
Grade 3 



September 
AS 
BS 
IS 



May 
BS 
CS 
It 
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GENERAL INFORMATION . * 

Title: California Achievement Test 
Authors: Ernest W. Tiegs, Willis W. Clark 
Publisher: CTB/McGraw-Hill, Monterey, CA 
Publication date: 1970 

Forms: There are two forms — A & B. The two forms were equated using the 
equipercentile method. 



Reading 

Vocabulary 
■ Computation 

Math 

Comparison- 
Concepts & Problems 

Language 
Auding 
Mechanics 
Usage & Structure 
Spelling 



Level 1 (Gr. 1.5-2 ) 

Items Time 



Level 2 (Gr. 2-4) 



92 
24 



^0 
47 



15 
38 
20 
20 



30 
16 



14 

17 



6 
10 
11 
10 

114 



Items 

40 

45 



72 
45 



66 
25 
25 

318 



Total Battery 296 

Manua4p. and Other Technical Aids: 

(1) Examiner's Manual (87 pp.) 
. (2) Test Coordinator ' s- Handbook (49 pp.) 
, (3) Bulletin of Technical Data (occasionally published) 



Time - 

13 
27 



26 
2? 



22 
6 
7 

123 



PRACTICAL CONSIDERATIONS 



Costs for Levels 1 & 2 



ERIC 



Complete battery (Reading, Math,. & Language) MS 
Complete battery (Reading, Math, & Language) HS 
Machine s cor ing/s tuden t 

Administration and Timing 



$18.80 pkg/35 
9.90 pkg/35 
.55 



• All tests on the CAT 70 are timed. The suggested testing schedule for 
administering the Complete batteries for Levels 1 and 2 is as follows: 



Level 1 (239 minutes) 
Level 2 (280 minutes) 



4 mornings 
4 mornings 
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FORMAT AND LAYOUT 

The format of the CAT is well planned and appealing. The reading 
subtests are especially well laid out and uncrowded. Illustrations 
are clear and type very readable. All answers are multiple choice 
on all subtests which may cause difficulties for the youngest children 
on certain subtests. On Math Computation, for example, the child must 
not only compute the answer, but must then find and mark the. correct 
answer from among four alternatives. In reality, th-'.s is a fairly 
sophisticated test-taking skill. For purposes of machine scoring, 
having the child code* his answers in such a manner is extremely 
convenient. Perhaps it cannot be avoided. 



ITEMS AND THEIR COVERAGE 

Content objectives for test items are clearly outlined in the 
Coordinator's Handbook, As with most achievement ' tests, CAT item 
coverage was guided by .a review of textbooks in 'reading, mathematics, 
and language used in the various states and a study of recommended 
curricular objectives also sampled from various sections of the 
country. Items were chosen to reflect "curricular relevance," 
difficulty, and ability to discriminate between high and low scores. 



CONSISTENCY 

Kuder^Richardson reliability coefficients reported for math and 
reading subtests on both Level 1 and Level 2 batteries computed at 
4 grade levels (1,6 to 4,6) are all at or above ,90, except for 
Reading Comprehension at the 1,6 grade level. 



' VALIDITY 

The CAT 70 has convincing face validity which is supported by the 
history of its development. For our purposes, the validity of the 
CAT can be confirmed by matching curriculum objectives of individu- 
alized programs in our study samples with the content objectives 
for each of the CAT subtests. 



SCORES AND NORMS 

203,684 students 
36 sta^tes 

P.S. stratified by: 

1. Geographic region 

2. Average enrollment /grade 

3. Community type 
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Minority group representation was not included in the stratification 
specifications. The authors, however, assert that there is adequate 
minority group representation in the sample insofar as minority 
groups participate in public education in the U.S. 



ADMINISTRATION AND SCORING 

No special training is necessary for teachers to administer the CAT. 
The Examiner's Manual for each level is \<fell written and provides 
explicit instructions. Three testing sessions varying in length from 
45 minutes to 1 hour are required to administer the entire battery of 
both Level 1 and Level 2. Tests may be hand or machine scored. 



COMMENTS 

Suggested schedule for using the CAT in the proposed study: 



September May 

1. ' CAT lA CAT iB 

2. CAT 11^ CAT 11^ 
3- CAT 11^ CAT 11^ 
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GENERAL INFORMATION . ' . 

-! ■■ ;f 

Title: Metropolitan Achievement Test . • 

Authors: Walter W. Durost, Harold H. Bixler, J. Wayne Wrightstone, 
George A. Prescott, Irving H. Balow 

Publisher: Harcourt Brace Jovanovich 

Publication date: 1971 ' ' 

Forms: Primer (Graded K.7 - 1.4) 
Form F (1971) 

Form H (not available for review) « 

Primary I (Grades 1.5-2.4) 

Form F (1970, 16 pages) 

Form G (not available for review) 

Form H (not available for review) 

Primary II (Grades 2.5 - 3.4) 
Form F (1970, 20 pages) 
Form G (1971, 20 pages) 
Form H (not available for review) 

\ 

Manual and Other Technical Aids: 

(1) Teacher's Handbook 

(2) Teacher's Directions * 

(3) Manual for Interpreting (128 pages) ^ 

Subtests: 

Primef 

(1) Listening for Sounds - "39 items measure pupils' knowledge of 

beginning and ending sounds and sound- 
letter relationships." 

• (2) Reading - "33 items measure pupils*' beginning reading skills." 

(3) Numbers - "34 items measure pupils' understanding of basic , 
mathematical principles and relationships." 

(Teacher's Directions [Primer], 1971, p. 3) 

Primary I 

(1) Word Knowledge - "35 items measure extent of pupils' reading ■ ' 

vocabulary. " 

(2) Word Analysis - "40 items measure pupils' knowledge of sound- 

letter relationships or skill in decoding." 

. (3) Reading - "42 items measure pupils' comprehension of written 
material. " 
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(4) Mathematics - 

"Part A: Concepts - 35 items measure pupils' under- 
standing of basic mathematical principles and 
relationships/* 

•■•Part B: Computation - 27 items measure pupils' ability 
to add and subtract one- and two-digit numbers wi,th no 
regrouping/' 

(teacher's Directions [Primary I], 1970, p. 3) 

Primary I.I 

(1) Word Knowledge - "40 items measure extent of pupils' reading 

V vocabulary." 

(2) Word Analysis - "35 items measure pupils' knowledge of sound- 

letter relationships or skill in decoding." 

. (3) Reading - "44 items measure pupils'^ comprehension of written 
material. " 

(4) Mathematics: Computa^on - "33 items 'measure pupils' ability 

to compute;," 

(5) ' Mathematics ; Concepts - "40 items measure pupils' under- 

standing of basic mathematical principles." 

(6) Mathemsttics: Problem Solving - "35 items measure pupils' 

ability to apply knowledge in solving numerical 
problems . " ' 

(Teacher's Directions [Primary II], 1970,., p. 3) 



PRACTICAL CONSIDERATIONS 



Costs* for Primer 




MRC Machine Scorable Edition (F,G) 
NCS^'Machine Scorable Edition (F) 
Primer Balttery (F,G) 
Stencil key for test booklet 
Teacher's Directions 
Teacher ' s Handbook 
Practice Page 
^Effective January 1, 19 75 



$16.50 pkg/35 

18.45, pkg/35 

11.25 pkg/35 

1.60 each 
.85 
.60 
.10 
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Costs* for Primary I 

MRC Machine Scorable Edition (F,G,H) 
NCS Machine Scorable Edition (F) 
Primary I Battery (Hand scored - F,G,H) 
Stencil key for test booklet (F,G,H) 
Teacher's Directions 
Teacher's Handbook 
^Effective January 1, 19 75 . 2 ^) 



$16.50 pkg/35 

18.40 pkg/35 

11.25 pkg/35 

1.85 each 
.85 
.60 
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Costs* for Primary II 

MRC Machine Scorable Edition (F,G,H) 
NCS Machine Scorable Edition (F) 
Primary II Battery (Hand scored - F,G,H)/ 
Stencil key for test booklet (F,G,H) 
Teacher's Directions 
Teacher's Handbook 
^Effective January 1, 1975 

^ Scoring Costs * ^ 



Primer 

Machine Scored (MRC Booklet) 
Hand Scored 

Primary I 

Machine Scored (MRC Booklet) 
Hand Scored 

Primary II 

Machine Scored (MRC Booklet) 
Hand Scored 



$18-25 pkg/35. 
19,25 pkg/35 
13.00 pkg/35 
2.50 each 
• 85' 
.60 



$/.50 
A. 75 



$ .60 
2.00 



$ .65 
2.25 



*These per pupil costs are effective January 1, 1975. 
All subtests of the MAT are timed. 

-J 

Primer 



Reconnnended 
Sittings 

1 

2 - 

3 

4 

5 

6 

7 

8 



P,art 

Practice Page 
Listening (pages 2 & 3) 

(page 4) 
Reading (page 5) 

(pages 6 & 7) 
Numbers (pages 8 & 9) 

(page 10) 

(page 11) 



'Administration Time 
in Minutes. 

ID 

20 • ' 

5 

5 
15* 

5 

5 

15* 

Total 80 minutes 



*These parts are not .teacherrdictated. 

^ It is recommended that the test be administered to groups of not 
more than 15 pupils. Two settings may be scheduled for one day 
provided there is a sufficient break between sittings. The whole 
battery should not be administered in one day. 
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Primary I 



Recommended 
Sittings 



2 
3 



Subtest 

Word Knowledge 
Word Analysis 
Reading 
Total Math 



Administration Time 
in^ Minutes 

. 15 
15* 
30 

30* 

Total 90 minutes 



*Portions teacher dictated, therefore, estimated times. 

It is recommended that the Primary I be administered in at 
least 3 sittings with a sufficient break between sittings. 
"Word Knowledge" and "Word Analysis" may be combined for a 
sitting (with a, break between them, of course) and "Reading" 
affd "Total Math" "as the second and third sittings. The 
complete test should not be administered in one day . 



Primary II 



Recommended 
Sittings 



2 



4 
5 



Subtests 

Word Knowledge 
Word Analysis 
Reading 

Jfath Computation 

Math Concepts 

Math Problem Solving 



Administration Time 
in Minutes 

18 

15* 

30 

18 

20* 

25 

Total 126 minutes 



*Portions teacher dictated, therefore, estimated times. 

The authors of the Primary II test recommend that it be 
administered in 5 sittings . Two sittings may be adminis- 
tered in one day if a sufficient break in time is provided 
between them. The complete test should not be administered 
in one day. 



FORMAT AND LAYOUT 
Primer 

\ In general, the format and layout of th*e Primer battery are appealin-g. 
^ However, page 4 in the "Listening for Sounds" subtest seems a bit 
overcrowded. 

\ 

The items are consistently numbered from left to right. Finally, the 
drawings used in the test are recognizable. 
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Primary I 

^ The physical layout of the Prim&ry I test is well designed. When 
there are two or more items in a row, they are numbered from left 
to right. Otherwise, the items are numbered down the page. The 
directions and samples are clear and concise. 

Jhe drawings used in the test, are good and the printing is excellent. 
Primary II 

The Primary II test is also designed well. The only complaint 
that might be raised is with the number of items per page in the 
mathematics subtests . 



ITEMS AND THEIR COVERAGE 

The development of the test^tems' for the 1970 edition of the 
Metropolitan Achievement test's began with a "curricular analyses." 



Five sources of information were used to decide what was being 
taught in* schopis in the U.S. Th^se were: (1) textbook series, 
(2) curricular syllabuses from individual school systems, (3) state- 
ments of objectives from various state and national committees..., 
(4) published summaries of curricular content in certain ^reas^..., 
and (5) statements by curricular experts... (Manual for Interpreting, 
1973, p. 18) > 

Item difficulty for the Primer, Primary I, and Primary II tests ranged 
from about .20 to .90. The average difficulty for th6 items on these 
tests was approximately .60. The items within each test are roughly 
arranged according to difficulty, i.e., easiest to most difficult. 

An item pool of abatrt* 12,000 items was used in the original, tryout 
of the battery (five forms at five levels). Aboiit half of- this 
number (6,000) was used, in the final version of the battery. 



jctiojgf' procedure to those "items answered correctly by progres- 
;ly higher percentages of pupils at successively higher grades," 



Although the authors specify that p-reference was given in the item 
selec 
sive] 

a 'table of. means for each test for each grade level of the norming 
population was not included. This perhaps would be the most prac- 
tical clieck of* how reasonable the difficulty level of each test and 
subtest is and whether the selection procedure* above p.^oved satisfactory. 



Primer 



The "Primer's three sections test types of skills that are reasonable 
to expect of entering first graders. There are, no items that seem to 
be biased against any cultural group. 

The Listening section is intended to measure students.^ knowledge of 
beginning sounds and ending sounds and sound-letter relationships. 
The Reading subtest requires the pupil to identify letters, to select 
the word that describes a picture and to choose one of .three* easy 
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sentences that describes a picture. The Numbers subtest measures a 
child's ability of counting, measurement, numerical recognition and 
his/her ability to add and subtract one-digit numbers. 

Primary I and Primary II 

The divisA$r of the reading content area into three subtests is very 
appealing; Reading primarily consists of decoding ability (word 
analysis subtest), the development of a good reading vocabulary 
(word knowledge subtest) and the ability to comprehend a written 
passage (reading subtest). The items of these subtests cover a 
wide variety of topic areas and seem to be applicable to children 
of all backgrounds. 

Math content is broken into two parts in the math subtest of the 
Primary I test, concepts and computation. The concept itetns test 
a wide range of mathematical abilities such as counting, knowing 
ordinal positioning^ \ time, place value, simple word problems, and 
more - less. All of these concepts are reasonable to test at this 
grade level. The computation items sample from addition and sub- 
traction problems (one and two digit) and verticle and horizontal 
format. A few problems with three addends are also included. 

Math content in the Primary II test consists of the above-mentioned 
subt?ests as well 'as a subtest entitled "problem solving." The prob- 
lem solving subtest contains word problems. Breaking mathematics 
into three subtests at this grade level is conceptually satisfying. 
The item coverage in each of these math areas is thorough. The 
student is required to perform on a variety of pertinent tasks. 



CONSISTENCY 

Each subtest "was designed to have an inteifnal consistency reliability 
^ -of about .90 for normal groups at eafeh singel grade level in the grade 
range intended for the battery." ( Manual for Interpreting , 1973, p. 21) 
Two reliability coefficients were reported as well as the standard 
error of measurement for each subtest. The reliability coefficients 
,are: 

(1) Split-half (odd-even) coefficient, corrected by the 
Spearman-Brown formula . 

(2) Saupe's estimate of Kuder-Richardson Formula 20. 

Note that the values for the internal consistency estimates and 
standard error of measurement are very reasonable for all subtests 
of the batteries. 
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Primer Battery (Form F) 



/ Test 






(Raw Scores) 


Listening for Somds 


.93 


.91 


■ 2.3 


Reading (Items 1-28) 


.93 


.89 


2.1 


Reading (Items 1-33). 


.93 


.90 


2.2 


- Numbers 


.96 


.93 


1.9 



*Saupe*s estimate of KR-20. 
**Split-half estimates. 



Primary I (Form G) 



Test . 


^ke* 


r ** 


^^mes 
(Raw Scores) 


— t 

Word Knowle^dge 


';88 


.94 


1.7 


" Word Analysis 


.90 


.94 


2.0 


Reading 


.95 


.96 


2.2 


Total Reading (W.K. & Rdg) 


.96 


.97 


2.8 


Total Mathematics 


.93 


.96 


2.4 



*Saupe's estimate "Of KR-20. 
**Split-half estimates. 
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Primary II (Form G) 



Test 


/ 

^ke* 


tt 


* 

S^es 
(Raw Scores) 


Word Knowledge 


.93 


.95 


2.0 


Word Analysis 


.90 


.93 


2.0 


Reading 


'. .93 


.95 


2.3 


Total Reading (W.K. & Rdg) 


' -':96 


.97 


3.1 


Math . Computation 


■ .86 


.91 


1.8 


Math. Concepts 


.85 


.89 


2.2 


Math. Problem Solving 


■ .88 


.92 


1.9 


Total Mathematics 




.96 


3.5 


(Comp. + Cone. + P.S.) 










*Saupe*s estimate of KR-20. 
Q **Split-half estimates. 
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VALIDITY 

The validity of the tests is only briefly mentioned in the Teacher's 
Handbook for both tests. However, the Manual. for Interpreting 
logically defends the content validity of each subtest of the batteries 
of the test. Two specific questions are 'addressed; "(1) content 
areas and objectives covered by the tests," and "(2) representative- 
ness of the content and objectives..." ( Manual for Interpreting , 1973, 
p. 25). A third question concerning content validity, the appropriate- 
ness of the test on the local level, is left for the consumer to con- 
sider. In general, the authors' coverage of this type of validity is 
excellent. 

Correlation data between the subtests of the Primer, Primary I, and 
Primary II batteries and the Otis-Lennon Mental Ability test are also 
presented. For the Primer, these were .55 for the Listening subtest, 
.52 for the' Reading subtest, and .63 for the Numbers subtest. For vthe 
Primary I battery these ranged from .54 to .65. For the Primary II 
battery the range was .48-. 72 with all but two coefficients equal to 
or greater than .62.* The number of pupils in the norming sample for 
this data was 4,000 to 7,000.^ 



SCORES AND NORMS 

Five major variables were used in selecting the norming population. 

(1) Socioeconomic index (median family income and median years of 
schooling of persons over the age of 24 in the community). 

(2) Size of the community. 

(3) Geograpjiic region,. ^ 

(4) Public versus non-public school system. 

(5) Mental ability test scores. 

The Manual for Interpreting describes in detail the selection and 
description of the norming sample. In addition, a more detailed 
. statistical description is contained in "Special Reporfs Noa. 7 
and 8" published by Harcourt Brace Jovanovich, Inc. 

The standardization took place at two times during the school year, 
in the fall (October, 1969) and in the spring (April, 1970). Norm 
conversion tables are contained in each test's Teacher's Handbook. 



ADMINISTRATION AND SCORING 
— ■ 1 

No special training other than studying the Teacher's Handbook and 
Teacher's Directions is needed in order to administer the MAT. 

The materials needed for the test are soft lead pencils (No. 2) , a 
"Testing — Do N»..n Disturb" sign for the door, a watch or clock with a 
second hand, a test booklet for each student and one for the adminis- 
trator as well as a copy of Teacher's Directions for the administrator. 

Fqr the machine scorable booklets, certain information needs to be 

coded by the administrator, e.g., student's nam'e, date of birth^ and sex. 
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Hand scoring the MAT is straightforward. However, if a large number 
were to be hand scored, it could prove to be very cumbersome and time- 
consuming. 

Hand scoring of this or any standardized test must, be monitored . 
closely in order to insure accuracy of the results. Quality control 
for the Evaluation Project (LTIDC) consists of a person scoring the test, 
another checking at least every other page of those tests previously 
scoi:ed, and finally, a check for errors in scoring by the data coder. 



COMMENTS 



This reviewer found no major flaws in the content and construction of 
the Primer, Primary 1, and Primary 11 Metropolitan Achievement Tests. 
It is technically well designed and intuitively appealing. 



GENERAL INFORMATION 



Title: SRA Assessment Survey: Primary Edition of the Achievement Series 
Authors:- Robert Naslund, Louis P. Thorpe, D. Weltz -.Lef ever 
Publisher: Science Research Associat-es 
Publication date: 1972 

Forms: Primary I (Grades 1.0-2,5) . 
. Form E (1972, 8 pages) 
Form F (1972, 8 pages) 

Primary II (Grades 2.5-4,0) 
Form E (1972, 8 pages) 
Form F (1972, 8 pages) 

Manual and Other Technical Aids: 

(1) Examiner's Manual 

(2) Technical Brief (14 pages) 

(3) Technical Report (118 pages) 

(4) Using Test Results (76 pages) y ' 

Subtests: . . 

Primary I 

(1) Reading (43 items) 

i/ord-Picture Association 
Sentence-Picture Association 
Comprehension 
Vocabulary 

(2) Mathematics (53 items) 

Concepts 
Computation 

Primary II 

(1) Reading (52 items') 

Reading Comprehension 
Vocabulary 

(2) Mathematics (58 items) 

Coacepts 
Computation 
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PRACTICAL CONSIDERATIONS 



Costs* for Primary I 



Ptiraary I Reading 
•Primary^ I Mathematics 
Practice Sheets 



Costs* for Primary II 



Primary II Reading 
Primary II Mathematics 
Practice Sheets 



$3.50 pkg/25 
3.50 pkg/25 
1.00 pkg/25 



$3.50 pkg/25 
3.50 pkg/25 
1.00 pkg/25 



Quantity prices: 20-199 pkg, each $3.15 
200 or more, each $2.98 

*Prices effective August , 1974 

Each package contains hand-scorahle test booklets /answer sheets 
for 25 students, one Examiner's Manual, and a Growth Scale Chart, 
A copy of the Conversion Table 'Booklet, including a Rights Key, 
is shipped with each order. 



Scoring Costs * 



The Primary I and Primary II tes*t booklets are consumable, i.e., 
they also serve as the answer sheets. In order to .use the SRA 
scoring service, these booklets are* rented. That is, for one 
price per student all of the testing materials (e.g^. , test book- 
lets, iixaminer's Manual, User's Manual, Technical Report, etcO 
are provided. In addition, SRA will score the tests. Three types 
of scoring services "are available with the base price of $1.11 
per student ($27.75 per class of 25). This includes national 
norms, grade equivalents, and raw scores. More elaborate analyses 
can be requested at additional costs. 

^Effective August 30, 1974. 

The subtests of. the Primary I and Primary II are not timed . 



Primary I 



Recommended 
Sittings 



3 
4 



Part 

Reading 

Word-Picture Association 

Sentence-Picture Association 

Comprehension 

Vocabulary 
Mathematics- 

Concepts 

Computation 



Administration Time 
in Minutes -(Estimated)* 



5 
5 
25 

15. 



20 



2Gi 



Total 105 minutes 
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Primary II 



Recommended 
Sittings 



2 
3 



Part 

Reading 

Comprehension 

Vocabulary 
Mathematics 

Concepts 

Computation ' 



Administration Time 
in Minutes (Estimated)* 



25 
15 



35 
20 



Total 95 minutes 



*The times listed are the suggested amounts of time each 
section is expected to take. The authors suggest a 

* testing period, until at least 90 percent of the students 
have finished- 
Two testing sessions a day are recommended. 



FORMAT AND LAYOUT 

The reading, and mathematics sections of the Primary I and Primary II 
tests arie separate forms. This allows for ease in administration. 

Overall, the' tasks are well .designed. The print is clear and the 
drawings are well done. 

However, two criticisms can be raised^ First, F6rm E of both the 
Primary I and Primary II tests is printed with brown ink. The print 
on th.' reading portion of the Primary I test which was reviewed 
was light which made this subtest difficult to read. The other ^ 
subtests printed in brown are, on the other hand, easily readable. 
Second, the weight of the paper on- which the tests are printed is 
too light. Because both sides of the page are used, the print on 
the reverse side of the pages shows through. 



ITEMS AND THEIR COVERAGE 

The development of the test items for the Primary tests included: 
an examination of basal texts which "account for those used in an 
estimated 75 percent of the classrooms in the United States"; 
examination of supplementary information supplied by the text 
publishers; and curriculum guides published by boards of education. 
These were supplemented by review of other testing programs and a 
telephone survey of curriculum specialists in the state departments * 
of boards of education, large cities, and smaller cities. 

The content of the ^nathematlcs subtests of the Primary I and Primary II 
tests covers a wide/ and representative range of content in an interesting 
manner. The dlvisi/on of the content into concepts and computation is 
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reasonable. The concepts section of the Primary I test has four 
parts: (1) sets and numeration, (2) operations, (3) measurement, 
and (4) place value* The computation section tests the students 
in the addition and subtraction of one- and two-digit numbers. 
The concepts section of the, Primary II test has five parts: 
(1) sets and numeration, (2) operations, (3) measurement and 
geometry, (4) place value, and (5) problem solving. The computation 
of the Primary II section tests the students in addition, subtraction, 
multiplication, and division. 

The reading portion of the Primary I and Primary II tests is divided 
into two sections, comprehension and vocabulary. The comprehension 
section of the Primary I includes: choosing the one picture of three 
presented that is best described by a given sentence; reading a short 
story and answering items 'that ask for restatement of the material, 
gathering information and reading beyond the story. The Primary II 
comprehension section also has items that ask the student to restate 
what s/he has read and to gather information in a passage. In addi- 
tion, the student is also asked to summarize and Sequence information 
and draw conclusions from short passages. 

There are two faults that this write? has found with the reading 
portion of the Primary I and Primary II tests. First, the passages 
in the comprehension subtests are long. If they were shorter, a 
greater variety of content could be included. The second criticism 
is more critical. Neither of these t^sts includes a subtest on 
decoding, e.g., sound-letter relationships, disc;^imitiation of parts 
of words (beginning, median, ending).; The ability to decode, which 
is essential to reading, probably should be tested in the Primary II 
test but most definitely should be covered in the primary I test. 



CONSISTENCY 



Kuder-Richardson reliability estimates (KR-20) wfere reported for both 
forms of the Primary I and Primary II tests at three grade ^Levels. 
(Reliability estimates were reported for three grade level^ because 
norming data is also reported for them.) 



Primary I 



Grade 1 Grade 2 Grade 3 



Form E 



Reading .88 ;.94 .93 

Comprehension « 79 ,89 .8^ 

Vocabulary .81 / ^89 .86 

Mathematics .'89 ' .91 .86 

Concepts .82 / . 83 .77 

Computation ' .83 / .88 .78 
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Primary I 






Grade 1 


Grade 2 


Grade 3 


Form F 








Kea Cling 




.93 


.92 


Comprehension 


.73 


.86 


.87 


Vocab ulary 


. 72 


. 89 


.86 


•naunema uxcs 


O "7 
• O/ 


on 

. 89 


.90 


. Concepts 


. 79 


.79 


.85 


Computation 


.80 


.87 


.82 






Primary 11 






Grade* 2 


Grade 3 


Grade 4 


Form E 








Keading 


.91 


.93 


.94 


Comprehension 


. 85 


. 87 


.89 


Vocabulary 


.84 


' .87 


.90 


Mathematics 


.88 


.91 


.92 


Concepts 


.83 


' .86 


.87 


Computation 


.76 


.85 


.86 



Form F 

Reading 

Comprehension 
Vocabulary 

Mathematics 
Concepts 
Computation 



/ 



.90 
.81 

.85 

.89 
.84 
.79 



.92 
.85 

.88 

.92 
.87 
.87 



.94 
.89 
.91 

.91 
.88 

.84 



These reliability estimates are reasonable,. The estimates for standard 
error of measurement (SEmeg) for raw scores are listed below. 



Primary I 



Grade 1 Grade 2 Grade 3 



Primary II 



Reading 
Mathematics 



2.79 
3.08 



2.58 
2.81 



1.99 
2.14 



Grade 2 

2.83 
3.06 



Grade 3 

2.70 
2.90 



Grade 4 

2.21 
2.29 



These are also reasonable for the number of items on the tests. 



VALIDITY 



ERIC 



The -authors only talk about the content validity of the achievement 
tests. .They state that thay relied on curriculum specialists in 
making decisions about the test content.* The final validity decision 
however, is placed on the test consumers. This information about 
validity was found iu the pamphlet "Using- Test Results.'' This lack 
of validity information is perhaps the major fault of the tests. 
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SCORES AND NORMS 

The standardization stud;^ for the achievement tests was conducted 
in April, 1971. -This study involved usable test results of 155,567 
students from 816 schools in 224 school districts, , selected according 
to a three-stage random sampling plan. The first »phase of this sam- 
pling plan divided the country into nine geographic regions (following 
the guidelines of the U.S. Bureau of Censiis) and selected 224 school 
districts from these nine strata. The se(^ond phase of the sampling 
procedure categorized the school districts randomly selected in 
Phase One according to six catagories: 

(1) All schools - . \ . • 

(2) Large city schools 

(3) Title I schools 

. (4) High SES schools 

(5) Rural/small town schools 

(6) Nonpublic schools. 

One school in each category which had a Grade 12 and one school which 
did not have a Grade 12 was then selected. In the third phase, class- 
rooms were randomly selected from each school. 

This sampling procedure allowed norm information to be computed for 
each of the six catagories listed in Phase Two. Users can request 
comparisons with any of these popjulations when SRA scoring is used, 
a very positive aspect of the test. In addition, norming data is 
provided for three grade levels for each test. 

(NOTE: The above information was extracted from the Technical Report . ) 



ADMINISTRATION AND SCORING 

Administration and scoring of the Primary I and Primary II test seems 
simple and straightforward. 

The materials that are needed for administration are: a practice 
sheet and test booklet for each student, standard pencils (No. 2), 
a copy of the test booklet and Examiner's Manual for the test admini- 
strator, and a clock or watch. 

If the booklets are to be machine scored, certain information must be 
hand coded by the administrator. If hand scoring is to be used, quality 
control measures should be provided. 



COMMENTS 

On the plus side of this test is the unique scoring alternatives which 
use the excellent norming procedure. 

On the debit side is: (1) the absence of validity information and 
(2) the lack of decoding items in the reading subtests. These are 
considered major .faults by this reviewer. no - 
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GENERAL INFORMATION 

Title: Stanford Achievement Test 

Authors: Richard Madden, Eric F. Gardner, Herbert C. Rudman, Bjom Karlsen, 
Jack C. Merwin , 

Publisher: Harcourt Brace Jovanovich, Inc., New York 

Publication date: 1973 

Forms: Primary I (Grades 1.5 - 2.4) 

Vocabulary (37 items) " 

Reading (87 items) 

Word Study Skills (60 items) 

Math Concepts (32 items) 

Math Computation and Application (32 items) 
Listening Comprehension (26 items) 

Primary II (Grades 2.5-3.4) 
Vocabulary (37 items) 
Reading (93 items) 
Word Study Skills (65 items) 
Math Concepts (35 items) 
Math Application (28 items) 
Spelling (43 items) 
Social Science (2 7 items) 
Science (27 items). 
Listening Comprehension (50 items) 

Primary III (Grades 3.5-4.4) 
Same as Primary 'II (above) 

There are three forms for each level: A, B, and C. 
Form C is i?ea>ding and math tests only. t 

Manual and other technical aids: 

1. Teacher's Guide for Interpreting (55 pages) 

2. Teacher's Directions for Administering (32 pages) 

3. Norms Booklet (24 pages) ^- 

4. Stanford Index of Instructional Objectives (21 pages) 



PRACTICAL COHBIDERATIONS 



Costs* for Primary Level I 



ERIC 



Primary Le^el I Battery (Hand scored, A & B) 
MRC Machine Scorable Booklet (A, B) ^ 
NCS Machine Scorable Booklet (A) 
Scoring service, per complete battery 

4 



2GG 



$11.25 pkg/35 
tjl5.95 pkg/35 
18.20 pkg/35 
.65 
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Costs* for Primary Level II 



Primary* Level II Battery (Hand scored, A & B) 
MRC Machine Scorable Booklet (A, B) 
NCS Machine Scorable Booklet (A) 
Scoring service, per complete battery 



$13.60 pkg/35 
18.25 pkg/35 
20.85 pkg/35 



.65 



Costs* for Primary Level III 




\ 



Primary Level III Battery (Hand scored, A & B) 
MRC Machine Scorable Booklet (A, B) 
UCS Machine Scorable Boqklet (A) 
Scoring service, per complete battery 



$14.50 pkg/35 
20.75 pkg/35 
23.75 pkg/35 



.45 



^Effective January 1, 1975. 



All subtests of the SAT are timed. 



Primary Level I — 4 hours 10 minutes (4 .days recommended) 
Primary Level II — 5 hours 40 minutes (6 days recommended) 



FORMAT AND LAYOUT ' 

\ 

The format is crowded in the Reading and Word Study Skills subtests; 
however, the illustrations and print are of acceptable quality. 



ITEMS AND THEIR COVERAGE 

Items in the 1973 edition of the SAT reflect the authors* analyses of 
the most widely used textbook series in the various subject area6, 
• a wide variety of courses of study, and the research literature per- 
taining to children's concepts, experiences, and vocabulary at 
successive ages or grades. The major goal of the authdrs was "to 
make sure that the content of the test would be in harmony with 
present instructional objectives and measure what is actually being 
taught in today's schools." A table of the jp values for each item 
at three grade levels is included in Part II of the Manual. No item 
was retained having a £ value less than .20. 



Reliability coefficients reported are greater than .85 for all Subtests, 
except for science and social science (.73 and .69, respectively). Two 
coefficients of reliability were computed for each subtest: 

1. Split-half corrected by Spearman-Brown Formula (rn) 

2. Kuder-Richardson Formula 20 




CONSISTENCY 
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VALIDITY 

Content validity of the SAT for our purposes can be determined by 
matching .the curriculum objectives of individualized programs in 
our study samples with the instructional 'objectives published for 
each of the subtests of the SAT. Apart from a discussion of the 
meaning of validity of an achievement test, no data supporting the 
SAT's claims to .validity are presented in the manual. 



SCORES AND NORMS 

Norms are bas4d on a restandardization of ^fehe SAT completed in 1973. 
This restand^dization was undertaken in part as a response to "the 
significant changes that had occurred in the elementary school curri- 
culum in the intervening yeais (i.e., 1964-1970)." The norming 
sample was drawn from 109 school systems in 43 states and finally 
included over 275,000 pupils. The norm sample attempted to be 
representative of the national population in terms of: ^" 

1» Geographic region 

2. Size of city 

3. Socio-economic status 

4. Public and non-public schools 

Blacks comprised 11.6% of the norm sample; Hispanic-Americans, 4.6%, 
accurately reflecting the proportions of these ethnic minorities in, 
the 1970 national population. More specific data is available in*^ 
the Technical Data Report, available upon request. 



ADMINISTRATIpN AND SCORING 

No special training is necessary for teachers to administer the SAT. 
The manual of administration is well written and explicit. A practice 
test is available to familiarize students with standardized test for- 
mat. A six half-day testing schedule is proposed in the manual. A 
substantial "Teacher's Guide for Interpreting" is available and is a 
useful document to help teachers utilize test results in their 
instructional planning. Machine scoring is available" from the 
publisher. 
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GENERAL INFORMATION 



Title: Animal Crackers, formerly Gumpgookies 
.Authors: Dorothy C. Adkins, Bonnie L. Ballif 
Publisher: CTB /McGraw-Hill, Monterey, CA 
Publication date: 1975 

Form: There is one form which' is designed for individual or small group 
administration in kindergarten and first grade. The test is com- 
posed of five scales and these are: 

(1) School Enjoyment 

( 2 ) Se If -Con f idence 

(3) Purposiveness 

(4) Instrumental Activity 

(5) Self-Evaluation 

There are twelve items on the test for each of these five scales, 
although factor analysis results indicate' overlap of items oyer 
scales." The test booklet consists of squares (six per page) each 
of which contains an identical pair of storybook-like animals. *As 
the child looks at these pictures, the examiner reads, for example 
"This 'bear (points to animal on left) shows its work to the teache 
"This bear (points to animal on right) hides its school work." 
Then says: "Now show me your bear." 



CONSISTENCY 

The reported Kuder-Richardson, formula 20 reliability coefficient for 
first grade samples was .98. Intercorrelations among the, components 
ranged from .76 to .92 for the first grade samples. ^ 



VALIDITY 



Earlier studies on Gumpgookies (mostly the same items as Animal 
• Crackers) have shown low positive correlations with age and intelli- 
gence, and significant relationships between test scores and teache 
ratings of motivation. Virginia Shipman (ETS) is ifsing Gumpgookies 
currently and finds that kindergarten scores predict significantly 
to third grade achievement scores. From a face validity standpoint 
the items are clearly stated, making minimal vocabulary demands and 
seem to cover well the whole area of "school adjustment." 
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APPLICATIONS 

Both SRI and Abt Associates used Gumpgookies in their evaluation 
studies of Project Follow Through. In the Abt study, there was 
generally low positive correlation between- Gximpgookies and various 
achievement and affective measures. Abt further found that Gump- 
goolcles scores are not normally distributed, giving evidence of a - 
ceiling effect. The SRI results also indicate some variation in 
responding as a function of ethnic group membership. CTB/McGraw- 
Hill has provided norms from a natural tryout of Animal Crackers. 
The sample size was 10,899 divided almost equally between first 
grad6 and kindergarten. This study indicated that black and 
Spanish-speaking children score lower than others in the test, 
but these differences are non-significant. 



ADMINISTRATION AND SCORING 

No special training is required for teachers to administer Animal 
Crackers.. The Examiner's Manual is well written and provides explicit, 
instructions. The test may be administered individually, taking 20- 
30 minutes per pupil or in small groups (requiring a longer instruction 
and practice session) taking 45-60 minutes. There are hand-scorable 
and machine'-scorable test booklets. The latter is on light-weight 
paper and pages would have to be backed x^ith an opaque sheet so that 
subsequent pictures don't show through. 

An Individual Performance Record and a Group Performance Record are 
available for hand-scoring and analysis. Norms are available for 
items, components, and overall scores from the CTB/McGraw-Hill 
national sample. 



PRACTICAL CONSIDERATIONS 
Costs 

Each package of 30 contains an Examiner's Manual; the hand-scorable 

edition alsp contains a Group Performance Record with Scoring Key. 



Code 

ANCR-M/S Test Books (Machine-scorable) $12.80 ,pkg/30 

ANCR-H/S Test Books (Hand-scorable) 11.00 pkg/30 

Accessories 

Additional Examiner's Manuals $ 1.50 ea. 

Administration Booklets ^ 3.75 ea. 

Individual Performance Records ' • 3.50 pkg/IOO 

Group Performance Records .10 ea. 
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Scoring ' 

Basic Scoring Service 

Optional Scoring Service 

Frequency Distrib^ion 
Administrator's Summary 



$ .60/student 



$ .08/student 
.03/student 

$25.00 minimum 
grade & level 
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